Nov 03 16:13:52.340868 ip-10-0-129-130 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:13:52.762848 ip-10-0-129-130 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:52.762848 ip-10-0-129-130 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:13:52.762848 ip-10-0-129-130 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:52.762848 ip-10-0-129-130 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768746 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768854 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768858 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768863 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768866 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.768858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768869 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768873 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768876 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768878 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768882 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768884 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768887 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768889 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768892 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768895 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768897 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768900 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768911 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768914 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768917 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768919 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768923 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768926 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768929 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768931 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.769093 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768934 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768937 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768940 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768942 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768945 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768949 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768951 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768954 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768957 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768959 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768962 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768965 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768968 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768970 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768975 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768981 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768984 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768988 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768991 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.769563 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768995 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.768998 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769001 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769003 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769006 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769009 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769019 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769023 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769025 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769028 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769031 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769033 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769036 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769039 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769041 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769044 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769047 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769050 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769053 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769055 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.770016 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769058 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769060 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769063 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769066 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769068 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769071 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769073 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769076 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769079 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769082 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769084 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769087 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769090 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769093 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769095 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769098 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769101 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769103 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769106 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.770489 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769115 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769118 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769121 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769123 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769126 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769129 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769132 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769134 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769137 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769140 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769142 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769146 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769150 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769154 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769156 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769159 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769162 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769165 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769167 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.770966 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769171 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769175 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769178 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.769181 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770073 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770081 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770084 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770087 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770090 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770093 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770096 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770098 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770101 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770112 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770115 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770118 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770121 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770124 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770126 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.771446 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770129 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770131 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770134 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770137 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770140 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770142 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770145 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770147 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770150 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770153 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770155 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770158 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770160 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770163 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770166 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770168 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770171 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770175 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770180 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.771941 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770183 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770186 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770189 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770192 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770195 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770197 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770200 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770203 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770206 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770209 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770212 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770215 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770218 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770220 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770225 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770228 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770231 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770234 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770237 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.772398 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770239 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770242 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770245 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770247 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770250 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770253 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770255 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770259 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770262 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770264 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770267 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770270 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770273 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770275 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770278 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770280 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770283 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770286 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770289 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770291 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.772877 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770294 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770297 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770300 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770303 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770306 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770308 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770311 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770314 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770316 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770319 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770321 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770324 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770327 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770329 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770332 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770334 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770337 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770339 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770342 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770345 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.773344 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770348 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770350 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770353 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770356 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770358 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770362 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770364 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770367 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770370 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770372 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770375 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770377 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770921 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770927 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770931 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770935 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770938 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770940 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770943 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.774433 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770946 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770948 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770951 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770953 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770956 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770959 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770961 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770964 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770966 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770969 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770972 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770975 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770978 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770981 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770984 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770986 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770989 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770992 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770995 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.770997 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.774927 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771000 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771002 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771005 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771008 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771010 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771013 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771015 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771018 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771021 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771024 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771026 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771029 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771032 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771034 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771037 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771039 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771042 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771044 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771047 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.775399 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771050 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771052 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771055 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771057 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771063 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771066 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771068 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771071 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771074 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771076 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771078 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771081 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771084 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771086 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771089 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771092 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771094 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771097 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771099 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771102 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.775857 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771104 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771107 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771111 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771113 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771116 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771118 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771121 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771123 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771126 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771129 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771131 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771133 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771136 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771138 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771141 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771144 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771148 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771152 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771156 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.776365 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771159 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771162 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771165 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771167 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771170 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771175 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771178 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771181 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771184 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771188 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771190 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771193 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771195 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771198 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771201 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771203 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771207 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771209 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771212 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.776837 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771214 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.771217 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772095 2583 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772106 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772112 2583 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772118 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772123 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772126 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772130 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772135 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772139 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772142 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772146 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772152 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772155 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772158 2583 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772161 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772164 2583 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772167 2583 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772170 2583 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772176 2583 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772178 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772181 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772185 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:13:52.777290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772188 2583 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772193 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772196 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772199 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772202 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772205 2583 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772208 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772211 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772215 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772218 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772222 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772225 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772228 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772231 2583 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772234 2583 flags.go:64] FLAG: --enable-server="true" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772237 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772242 2583 flags.go:64] FLAG: --event-burst="100" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772245 2583 flags.go:64] FLAG: --event-qps="50" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772248 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772252 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772255 2583 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772261 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772264 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772267 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772270 2583 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:13:52.777857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772273 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772276 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772279 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772282 2583 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772285 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772288 2583 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772291 2583 flags.go:64] FLAG: --feature-gates="" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772296 2583 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772299 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772302 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772306 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772309 2583 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772312 2583 flags.go:64] FLAG: --help="false" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772315 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-129-130.ec2.internal" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772320 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772323 2583 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772326 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772329 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772333 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772336 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772339 2583 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772341 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772344 2583 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:13:52.778476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772347 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772351 2583 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772354 2583 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772357 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772360 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772367 2583 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772370 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772373 2583 flags.go:64] FLAG: --lock-file="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772376 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772379 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772382 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772388 2583 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772391 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772394 2583 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772397 2583 flags.go:64] FLAG: --logging-format="text" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772400 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772404 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772408 2583 flags.go:64] FLAG: --manifest-url="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772411 2583 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772416 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772419 2583 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772423 2583 flags.go:64] FLAG: --max-pods="110" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772426 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772429 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772433 2583 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:13:52.779118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772436 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772439 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772442 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772445 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772453 2583 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772457 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772460 2583 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772464 2583 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772467 2583 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772470 2583 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772473 2583 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772476 2583 flags.go:64] FLAG: --port="10250" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772480 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772483 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0538baca60d541a1d" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772486 2583 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772489 2583 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772492 2583 flags.go:64] FLAG: --register-node="true" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772495 2583 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772499 2583 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772502 2583 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772505 2583 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772508 2583 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772512 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772515 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772518 2583 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772521 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:13:52.779730 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772524 2583 flags.go:64] FLAG: --runonce="false" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772527 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772531 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772533 2583 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772536 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772539 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772544 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772547 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772550 2583 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772553 2583 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772556 2583 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772559 2583 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772562 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772565 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772568 2583 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772572 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772591 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772596 2583 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772601 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772606 2583 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772610 2583 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772613 2583 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772616 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772619 2583 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772623 2583 flags.go:64] FLAG: --v="2" Nov 03 16:13:52.780383 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772628 2583 flags.go:64] FLAG: --version="false" Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772633 2583 flags.go:64] FLAG: --vmodule="" Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772638 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.772641 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772792 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772796 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772800 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772803 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772806 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772809 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772812 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772815 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772818 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772820 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772824 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772827 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772829 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772832 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772836 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772840 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.780997 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772842 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772845 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772851 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772854 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772856 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772859 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772862 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772865 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772868 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772870 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772873 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772875 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772879 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772881 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772884 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772887 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772890 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772893 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772895 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772898 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.781492 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772901 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772903 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772906 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772909 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772912 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772914 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772917 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772920 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772922 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772925 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772928 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772930 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772933 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772935 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772940 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772942 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772945 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772947 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772950 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772952 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.781994 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772955 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772957 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772960 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772962 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772965 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772967 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772970 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772973 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772975 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772978 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772981 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772984 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772986 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772990 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772993 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772996 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.772998 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773001 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773004 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.782469 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773006 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773009 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773012 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773015 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773017 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773020 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773022 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773027 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773029 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773032 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773034 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773039 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773043 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773046 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773049 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773052 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773054 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773057 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773059 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.782959 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773062 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773064 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773067 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773070 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773072 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773075 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773077 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773081 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773084 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773086 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773089 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773091 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.773122 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773162 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773166 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773170 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.783417 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773173 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773176 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773179 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773182 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773185 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773189 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773191 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773194 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773196 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773199 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773202 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773204 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773207 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773209 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773212 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773214 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773217 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773221 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773225 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.783817 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773228 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773231 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773234 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773237 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773240 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773243 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773245 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773248 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773250 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773253 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773256 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773258 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773261 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773264 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773266 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773269 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773271 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773274 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773278 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.784264 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773281 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773283 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773286 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773289 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773291 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773294 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773296 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773299 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773301 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773304 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773307 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773309 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773312 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773316 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773319 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773322 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773325 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773329 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773331 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.784834 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773334 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773337 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773339 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773342 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773344 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773347 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773350 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773353 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773356 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773359 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773362 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773365 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773369 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773371 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773374 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773377 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773380 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773382 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773385 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773387 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.785287 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773390 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773392 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773396 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773398 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773401 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773403 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773406 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773408 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773411 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773413 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773417 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773420 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773422 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773425 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773427 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773429 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773432 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773435 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773437 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773440 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.785832 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773442 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773445 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773448 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773451 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773455 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.773457 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.780807 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.780826 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.781440 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.781446 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781511 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781518 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781521 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781525 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781529 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781531 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781534 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781537 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781540 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.786319 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781542 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781545 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781548 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781551 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781554 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781557 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781559 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781562 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781565 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781567 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781570 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781572 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781575 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781590 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781593 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781596 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781598 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781602 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781605 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781608 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.786803 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781617 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781620 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781622 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781625 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781628 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781630 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781633 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781635 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781638 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781641 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781643 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781646 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781648 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781651 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781654 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781657 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781659 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781662 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781665 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.787276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781667 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781672 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781676 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781679 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781682 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781685 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781688 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781691 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781693 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781696 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781699 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781702 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781704 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781707 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781711 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781714 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781716 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781719 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781721 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.787742 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781727 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781729 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781732 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781735 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781737 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781740 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781742 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781745 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781748 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781750 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781753 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781756 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781758 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781761 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781763 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781766 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781769 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781771 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.788219 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781774 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781776 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781779 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781782 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781784 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781787 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781790 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781792 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781795 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781798 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781801 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781803 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781806 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781810 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781814 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781817 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781819 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781822 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781825 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.788713 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.781827 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.781858 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782018 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782023 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782026 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782029 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782032 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782034 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782037 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782039 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782042 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782046 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782048 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782051 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782054 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782057 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.789175 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782059 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782063 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782067 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782070 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782073 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782075 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782078 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782081 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782084 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782087 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782090 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782092 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782095 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782098 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782100 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782103 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782105 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782108 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782111 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.789726 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782114 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782116 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782119 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782122 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782124 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782144 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782148 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782152 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782156 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782159 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782162 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782165 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782167 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782170 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782173 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782176 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782178 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782181 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782184 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782186 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.790470 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782189 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782192 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782195 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782198 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782200 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782203 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782205 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782208 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782211 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782213 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782216 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782218 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782221 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782224 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782227 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782229 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782232 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782234 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782237 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.791142 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782240 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782242 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782245 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782247 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782250 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782252 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782255 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782258 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782262 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782266 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782270 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782272 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782275 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782278 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782280 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782283 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782286 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782289 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782291 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.792276 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782294 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782296 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782299 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782301 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782304 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782307 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782309 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782312 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782314 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782317 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782320 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782322 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782325 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782328 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782330 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.782353 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:52.792928 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782385 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782389 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782392 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782395 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782399 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782403 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782406 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782408 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782411 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782413 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782416 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782419 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782421 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782424 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782427 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782430 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782432 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782435 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782437 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:52.793374 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782440 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782442 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782445 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782448 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782450 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782455 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782458 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782461 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782463 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782466 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782468 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782471 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782473 2583 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782476 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782479 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782482 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782485 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782487 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782490 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782492 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:52.793892 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782495 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782497 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782500 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782503 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782505 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782508 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782510 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782514 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782516 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782519 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782521 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782524 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782526 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782529 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782531 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782534 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782536 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782540 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782543 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:52.794358 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782546 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782549 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782551 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782553 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782556 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782559 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782561 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782564 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782566 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782569 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782573 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782576 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782590 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782593 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782596 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782598 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782601 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782604 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782606 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:52.794822 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782609 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782612 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782619 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782622 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782625 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782628 2583 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782630 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782633 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782636 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782638 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782640 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782645 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782647 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782650 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782652 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782655 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782657 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782660 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782662 2583 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782665 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:52.795294 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782668 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782670 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782673 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782675 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782679 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782682 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782684 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782687 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:52.782692 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.782757 2583 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.784864 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.785759 2583 server.go:1014] "Starting client certificate rotation" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.785860 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.785892 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.788471 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:52.795779 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.793190 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:52.803092 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.803071 2583 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:13:52.806442 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.806425 2583 log.go:25] "Validated CRI v1 image API" Nov 03 16:13:52.807318 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.807291 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:13:52.809934 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.809909 2583 fs.go:135] Filesystem UUIDs: map[33c7b21c-c774-46ea-a28e-24b2a9c46627:/dev/nvme0n1p3 3c4d8ae7-1fe2-41d2-aadf-90f59d2fe2d9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 16:13:52.809980 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.809934 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 16:13:52.815096 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.814977 2583 manager.go:217] Machine: {Timestamp:2026-11-03 16:13:52.813706556 +0000 UTC m=+0.352654407 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086868 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28984d6ec630f475e4414e41de3dcd SystemUUID:ec28984d-6ec6-30f4-75e4-414e41de3dcd BootID:c86926e4-deb7-4436-bc6d-6de524aafa25 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:cb:c2:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:cb:c2:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:17:e7:32:e8:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:13:52.815096 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.815090 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:13:52.815234 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.815208 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:13:52.818696 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.818662 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:13:52.818849 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.818696 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:13:52.819443 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.819428 2583 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:13:52.819443 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.819439 2583 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:13:52.819559 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.819466 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:52.819559 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.819479 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:52.890108 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.890075 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:52.890254 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.890152 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xsmf" Nov 03 16:13:52.901290 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.901262 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6xsmf" Nov 03 16:13:52.951389 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.951364 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:13:52.954935 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.954919 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:13:52.955132 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955122 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:13:52.955187 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955159 2583 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:13:52.955187 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955187 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:13:52.955271 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955201 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:13:52.955271 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955212 2583 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:13:52.955271 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955230 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:13:52.955641 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.955625 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:52.957832 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.957810 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:13:52.962072 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.962059 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:52.963082 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.963068 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:13:52.963123 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.963100 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:13:52.964631 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964609 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:13:52.964716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964638 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:13:52.964716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964649 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:13:52.964716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964659 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:13:52.964716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964669 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:13:52.964716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.964684 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:13:52.965111 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965099 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:13:52.965155 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965116 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:13:52.965155 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965134 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:13:52.965155 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965144 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:13:52.965310 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965158 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:13:52.965310 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965172 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:13:52.965924 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965910 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:13:52.965965 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.965941 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:13:52.970743 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.970726 2583 server.go:1263] "Started kubelet" Nov 03 16:13:52.970844 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.970813 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:13:52.970929 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.970877 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:13:52.970970 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.970955 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:13:52.971485 ip-10-0-129-130 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:13:52.973199 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.973183 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:13:52.973815 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.973802 2583 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:13:52.977101 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:52.977078 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 16:13:52.978449 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.978427 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:13:52.978449 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.978436 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:52.978621 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:52.978530 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found" Nov 03 16:13:52.979374 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.979342 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:13:52.979465 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.979419 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:13:52.979465 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.979436 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:13:52.979694 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.979595 2583 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:13:52.979694 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.979603 2583 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:13:52.980015 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980000 2583 factory.go:153] Registering CRI-O factory Nov 03 16:13:52.980093 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980018 2583 factory.go:223] Registration of the crio container factory successfully Nov 03 16:13:52.980093 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980087 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:13:52.980195 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980097 2583 factory.go:55] Registering systemd factory Nov 03 16:13:52.980195 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980105 2583 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:13:52.980195 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980122 2583 factory.go:103] Registering Raw factory Nov 03 16:13:52.980195 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.980136 2583 manager.go:1199] Started watching for new ooms in manager Nov 03 16:13:52.981329 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.981311 2583 manager.go:319] Starting recovery of all containers Nov 03 16:13:52.992036 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.992011 2583 manager.go:324] Recovery completed Nov 03 16:13:52.998015 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:52.997993 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:53.001297 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.001275 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:53.001381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.001313 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:53.001381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.001324 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:53.001981 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.001965 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:53.002059 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.001967 2583 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:13:53.002059 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.002027 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:13:53.002059 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.002053 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:13:53.002792 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.002776 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:53.002861 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.002803 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:53.002861 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.002814 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:53.003299 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.003287 2583 policy_none.go:50] "Start" Nov 03 16:13:53.003330 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.003303 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:13:53.003330 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.003313 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:13:53.004314 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.004302 2583 policy_none.go:44] "Start" Nov 03 16:13:53.004512 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.004492 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:53.007487 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.007468 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:53.026697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.026635 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:53.026697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.026677 2583 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found Nov 03 16:13:53.026840 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:53.026734 2583 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-130.ec2.internal\" not found" node="ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.049250 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.049202 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:13:53.050597 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.050564 2583 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found Nov 03 16:13:53.053436 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.053420 2583 manager.go:341] "Starting Device Plugin manager" Nov 03 16:13:53.054041 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:53.054026 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:13:53.054041 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054040 2583 server.go:87] "Starting device plugin registration server" Nov 03 16:13:53.054295 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054282 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:13:53.054364 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054298 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:13:53.054643 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054460 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:13:53.054643 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054544 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:13:53.054643 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.054552 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:13:53.055106 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:53.055089 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:13:53.055177 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:53.055141 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-130.ec2.internal\" not found" Nov 03 16:13:53.103920 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.103745 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:13:53.105170 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.105152 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:13:53.105278 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.105177 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:13:53.105278 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.105201 2583 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:13:53.105278 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:53.105246 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 16:13:53.117034 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.117012 2583 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-130.ec2.internal" not found Nov 03 16:13:53.119081 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.119055 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:53.155420 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.155390 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:53.158761 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.158743 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:53.158856 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.158773 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:53.158856 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.158786 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:53.158856 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.158811 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.176433 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.176411 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.205559 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.205532 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"] Nov 03 16:13:53.206979 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.206963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.207946 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.207920 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.237395 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.237367 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.241381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.241366 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.263376 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.263347 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:53.263536 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.263350 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:53.380807 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.380710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.380807 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.380757 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:53.380807 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:53.380777 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801276 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801241 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801423 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801423 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801423 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801326 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801423 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801335 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.801423 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.801343 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.887543 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.887504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:13:54.887689 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:54.887573 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:13:55.106092 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.106011 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 16:13:55.106626 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.106167 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:55.106626 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.106203 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:55.106626 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.106238 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:55.275628 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.275592 2583 apiserver.go:52] "Watching apiserver" Nov 03 16:13:55.275777 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.275569 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:54 +0000 UTC" deadline="2028-07-16 07:40:04.103481466 +0000 UTC" Nov 03 16:13:55.275777 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.275710 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14895h26m8.827775966s" Nov 03 16:13:55.299222 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.299195 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:55.317372 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.317343 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:13:55.317668 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.317645 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-t6s9c","openshift-multus/multus-hwdlx","openshift-multus/network-metrics-daemon-qgkpn","openshift-network-operator/iptables-alerter-qb7s9","openshift-ovn-kubernetes/ovnkube-node-c84vd","kube-system/global-pull-secret-syncer-4qqx2","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r","openshift-network-diagnostics/network-check-target-zhgz9","openshift-multus/multus-additional-cni-plugins-qdl9p","kube-system/konnectivity-agent-h2qrh","openshift-cluster-node-tuning-operator/tuned-lht4j","openshift-image-registry/node-ca-4tjbw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal"] Nov 03 16:13:55.318883 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.318861 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.320039 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.320016 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.321399 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.321378 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.321526 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.321457 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:13:55.322348 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.322335 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.323533 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.323517 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.324707 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.324690 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.324780 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.324751 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:13:55.325749 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.325732 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.326798 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.326779 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:55.326873 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.326831 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:13:55.328160 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.328143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.329128 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.329111 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.330222 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.330206 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.331430 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.331412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.334103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.334081 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:13:55.334191 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.334163 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.334244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.334192 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.335024 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.334984 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.335024 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.334990 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.335146 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.335115 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:13:55.335187 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.335152 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:13:55.335285 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.335272 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:13:55.348132 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.348107 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:55.348258 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.348176 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-stpzs" Nov 03 16:13:55.363610 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.363590 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-stpzs" Nov 03 16:13:55.367496 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.367473 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45286eece609f171819d9469f4da982c.slice/crio-1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56 WatchSource:0}: Error finding container 1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56: Status 404 returned error can't find the container with id 1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56 Nov 03 16:13:55.367664 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.367651 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90d8c1e0bb8232281e68f4371e9fed3.slice/crio-171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205 WatchSource:0}: Error finding container 171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205: Status 404 returned error can't find the container with id 171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205 Nov 03 16:13:55.373118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.373099 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:13:55.400518 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.400497 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:13:55.403836 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.403815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.403944 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.403848 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.403944 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.403917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404028 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.403983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.404068 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.404114 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404062 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.404114 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.404186 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404117 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng4vr\" (UniqueName: \"kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404186 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxqrw\" (UniqueName: \"kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.404253 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.404253 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404223 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjcx5\" (UniqueName: \"kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.404316 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404290 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nql7\" (UniqueName: \"kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404326 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.404474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.404474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.404474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:55.404623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404579 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.404711 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhxv\" (UniqueName: \"kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.404711 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404649 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.404711 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404686 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404710 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404782 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.404843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404830 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404989 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404865 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.404989 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404989 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.404989 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.404979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.405117 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.405117 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.405186 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvqh\" (UniqueName: \"kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.405186 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405186 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.405321 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405188 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.405321 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405211 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.405321 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.405321 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405281 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.405474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405324 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.405474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405353 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.405474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405378 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb5t\" (UniqueName: \"kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.405474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405392 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405429 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405476 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405500 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405543 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405568 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405586 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405678 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405886 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405886 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.405886 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405787 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tphwm\" (UniqueName: \"kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.405886 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405823 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.405886 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405929 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.405982 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406005 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406070 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.406103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406111 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj497\" (UniqueName: \"kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406177 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406202 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406226 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406307 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406330 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406358 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406406 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.406483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406472 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.406866 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406498 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.406866 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406523 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.406866 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.406550 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.429295 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.429204 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"45286eece609f171819d9469f4da982c","Type":"ContainerStarted","Data":"1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56"} Nov 03 16:13:55.430366 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.430345 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"b90d8c1e0bb8232281e68f4371e9fed3","Type":"ContainerStarted","Data":"171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205"} Nov 03 16:13:55.435525 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.435495 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:13:55.435654 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.435629 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.435966 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.435953 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.436012 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.435968 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436243 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436261 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436301 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436305 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436313 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436335 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.436381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436333 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.436692 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436496 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:13:55.436692 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436576 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:13:55.436692 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436617 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:13:55.436692 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436616 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:13:55.436819 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436731 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:13:55.436819 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.436734 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:13:55.445728 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.445702 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:13:55.445728 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.445719 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.445728 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.445702 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:13:55.449040 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449017 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:13:55.449149 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449097 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:13:55.449200 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449168 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:13:55.449200 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449191 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:13:55.449284 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449192 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:13:55.449356 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449339 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:13:55.449904 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.449889 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:13:55.450030 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.450011 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:13:55.506861 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507004 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507004 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506899 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507004 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506928 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphwm\" (UniqueName: \"kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.507004 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506958 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507004 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.506967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507004 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507002 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507042 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507062 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507117 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507136 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507162 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507178 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507193 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507244 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.507369 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507379 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.507461 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:13:56.007420804 +0000 UTC m=+2.226505673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj497\" (UniqueName: \"kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507582 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507598 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507611 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507649 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507615 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507724 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507753 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507777 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.507865 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507816 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507828 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507911 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507910 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507910 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.507919 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508174 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508304 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508314 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.508629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508364 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508380 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.508501 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.508564 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:13:56.008544922 +0000 UTC m=+2.227629780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508574 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4vr\" (UniqueName: \"kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508761 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqrw\" (UniqueName: \"kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcx5\" (UniqueName: \"kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.509408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508867 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nql7\" (UniqueName: \"kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508872 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508894 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.508967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509021 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509054 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509050 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhxv\" (UniqueName: \"kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509203 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509210 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509238 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509266 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509320 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509355 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.510274 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509370 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509457 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509464 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509483 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509153 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509480 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509371 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509585 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509640 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509693 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509692 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509717 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509747 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509755 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509769 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509772 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509817 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509824 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvqh\" (UniqueName: \"kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509861 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509904 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.509957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510002 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510045 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510113 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510153 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.511907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510216 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb5t\" (UniqueName: \"kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510252 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510387 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510465 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510472 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510545 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510555 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.512740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510601 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510605 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510637 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510691 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510750 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.510792 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.511100 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.511155 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.511255 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.512533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.512543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.513040 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.513289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.513285 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.531085 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.531047 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:55.531085 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.531079 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:55.531085 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.531093 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:55.531350 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:55.531180 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:13:56.03115982 +0000 UTC m=+2.250244691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:55.531571 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.531545 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphwm\" (UniqueName: \"kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.531972 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.531949 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj497\" (UniqueName: \"kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.533152 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.533125 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqrw\" (UniqueName: \"kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:55.533352 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.533332 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4vr\" (UniqueName: \"kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.534170 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.533756 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvqh\" (UniqueName: \"kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.534170 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.533758 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhxv\" (UniqueName: \"kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.534170 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.534115 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb5t\" (UniqueName: \"kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.534330 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.534165 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcx5\" (UniqueName: \"kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.535279 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.535261 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nql7\" (UniqueName: \"kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.637223 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.637145 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:13:55.646322 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.646295 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6cd40ea_b486_4d7d_a8ec_49d153976a09.slice/crio-0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378 WatchSource:0}: Error finding container 0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378: Status 404 returned error can't find the container with id 0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378 Nov 03 16:13:55.653625 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.653604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hwdlx" Nov 03 16:13:55.661725 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.661700 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde7e318_0d30_4892_81f1_3e4f19c782f5.slice/crio-6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f WatchSource:0}: Error finding container 6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f: Status 404 returned error can't find the container with id 6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f Nov 03 16:13:55.679266 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.679236 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:13:55.688750 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.688722 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0811b831_8592_4677_aa31_cd58433a3f5c.slice/crio-4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5 WatchSource:0}: Error finding container 4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5: Status 404 returned error can't find the container with id 4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5 Nov 03 16:13:55.692277 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.692259 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:13:55.694961 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.694938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:13:55.700858 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.700836 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16e7255_2933_413f_9114_4b1e2f93d744.slice/crio-458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e WatchSource:0}: Error finding container 458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e: Status 404 returned error can't find the container with id 458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e Nov 03 16:13:55.701841 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.701821 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:13:55.702974 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.702955 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:13:55.704349 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.704327 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8555d4b_9866_49fa_b635_d9b8e26f8b3b.slice/crio-62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6 WatchSource:0}: Error finding container 62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6: Status 404 returned error can't find the container with id 62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6 Nov 03 16:13:55.708219 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.708203 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:13:55.711705 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.711679 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de08296_dbf4_45f3_8dc4_33cbf8de0658.slice/crio-16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a WatchSource:0}: Error finding container 16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a: Status 404 returned error can't find the container with id 16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a Nov 03 16:13:55.712432 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.712411 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebced67_f8f8_480e_a1bc_4add3059fd0e.slice/crio-3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1 WatchSource:0}: Error finding container 3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1: Status 404 returned error can't find the container with id 3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1 Nov 03 16:13:55.712727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:55.712713 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:13:55.716515 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.716493 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b87e4_de05_417f_82ba_53e481c27611.slice/crio-354a3ddcc1b3a65438b3511dfad12e401069ce09dff8367b0be369103e5af51f WatchSource:0}: Error finding container 354a3ddcc1b3a65438b3511dfad12e401069ce09dff8367b0be369103e5af51f: Status 404 returned error can't find the container with id 354a3ddcc1b3a65438b3511dfad12e401069ce09dff8367b0be369103e5af51f Nov 03 16:13:55.721755 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:13:55.721732 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda163d0db_7c08_45c9_bf83_6132d280a72a.slice/crio-8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f WatchSource:0}: Error finding container 8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f: Status 404 returned error can't find the container with id 8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f Nov 03 16:13:56.014280 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.014143 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:56.014280 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.014206 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:56.014493 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.014392 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:56.014493 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.014470 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:13:57.014435715 +0000 UTC m=+3.233520568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:56.014493 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.014532 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:56.014493 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.014573 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:13:57.014553774 +0000 UTC m=+3.233638654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:56.038180 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.038148 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:56.097952 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.097901 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:56.115090 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.115055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:56.116049 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.115225 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:56.116049 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.115253 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:56.116049 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.115265 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:56.116049 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.115321 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:13:57.115304737 +0000 UTC m=+3.334389602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:56.232797 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.232765 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:56.369229 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.366114 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:55 +0000 UTC" deadline="2028-05-08 21:29:15.616359508 +0000 UTC" Nov 03 16:13:56.369229 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.366177 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13253h15m19.250186708s" Nov 03 16:13:56.427241 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.426642 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:56.427241 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:56.426820 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:13:56.440394 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.440304 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" event={"ID":"a8555d4b-9866-49fa-b635-d9b8e26f8b3b","Type":"ContainerStarted","Data":"62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6"} Nov 03 16:13:56.441810 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.441743 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qb7s9" event={"ID":"0811b831-8592-4677-aa31-cd58433a3f5c","Type":"ContainerStarted","Data":"4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5"} Nov 03 16:13:56.443278 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.443211 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lht4j" event={"ID":"746b87e4-de05-417f-82ba-53e481c27611","Type":"ContainerStarted","Data":"354a3ddcc1b3a65438b3511dfad12e401069ce09dff8367b0be369103e5af51f"} Nov 03 16:13:56.458856 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.458818 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e"} Nov 03 16:13:56.481894 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.481801 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwdlx" event={"ID":"fde7e318-0d30-4892-81f1-3e4f19c782f5","Type":"ContainerStarted","Data":"6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f"} Nov 03 16:13:56.496648 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.496609 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6s9c" event={"ID":"f6cd40ea-b486-4d7d-a8ec-49d153976a09","Type":"ContainerStarted","Data":"0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378"} Nov 03 16:13:56.515302 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.515257 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tjbw" event={"ID":"a163d0db-7c08-45c9-bf83-6132d280a72a","Type":"ContainerStarted","Data":"8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f"} Nov 03 16:13:56.534784 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.534730 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h2qrh" event={"ID":"debced67-f8f8-480e-a1bc-4add3059fd0e","Type":"ContainerStarted","Data":"3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1"} Nov 03 16:13:56.549902 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:56.549862 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerStarted","Data":"16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a"} Nov 03 16:13:57.023591 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.023551 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:57.023758 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.023616 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:57.023758 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.023739 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:57.023879 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.023800 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:13:59.023781737 +0000 UTC m=+5.242866603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:57.024219 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.024197 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:57.024316 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.024264 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:13:59.024249037 +0000 UTC m=+5.243333901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:57.126014 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.125977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:57.126526 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.126137 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:57.126526 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.126156 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:57.126526 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.126171 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:57.126526 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.126242 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:13:59.126222867 +0000 UTC m=+5.345307731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:57.367125 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.367012 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:55 +0000 UTC" deadline="2028-07-18 03:47:17.453744686 +0000 UTC" Nov 03 16:13:57.367125 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.367054 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14939h33m20.086695359s" Nov 03 16:13:57.426704 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.425855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:57.426704 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.426044 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:13:57.426704 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:57.426518 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:57.426704 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:57.426650 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:13:58.430933 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:58.427570 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:58.430933 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:58.427701 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:13:59.043015 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.042931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:59.043015 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.042991 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:59.043228 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.043176 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:59.043278 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.043252 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:03.043231741 +0000 UTC m=+9.262316595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:59.043721 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.043694 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:59.043854 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.043761 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:03.043739994 +0000 UTC m=+9.262824863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:59.146230 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.144081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:13:59.146230 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.144279 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:59.146230 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.144302 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:59.146230 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.144314 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:59.146230 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.144371 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:14:03.144354529 +0000 UTC m=+9.363439393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:59.425765 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.425535 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:13:59.425998 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.425971 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:13:59.426082 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.425803 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:13:59.426184 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:13:59.426142 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:13:59.580774 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.580702 2583 generic.go:356] "Generic (PLEG): container finished" podID="45286eece609f171819d9469f4da982c" containerID="79ae7886b4439f3fc1f530d84ed615e72bc323e727c000d493d9062ec9f8204c" exitCode=0 Nov 03 16:13:59.581136 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.580774 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"45286eece609f171819d9469f4da982c","Type":"ContainerDied","Data":"79ae7886b4439f3fc1f530d84ed615e72bc323e727c000d493d9062ec9f8204c"} Nov 03 16:13:59.589466 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.587748 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"b90d8c1e0bb8232281e68f4371e9fed3","Type":"ContainerStarted","Data":"b6d7e346e25c50380a49b8464c4a39937f7244492738fc7a3b2db5d3f3ac95d9"} Nov 03 16:13:59.595639 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.595579 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="25518ec31b9a790137628fd39b6769632dc3febc2210704ed88a9575574fdc0a" exitCode=0 Nov 03 16:13:59.595639 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:13:59.595622 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"25518ec31b9a790137628fd39b6769632dc3febc2210704ed88a9575574fdc0a"} Nov 03 16:14:00.426690 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.426654 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:00.426884 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:00.426796 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:00.601046 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.601009 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h2qrh" event={"ID":"debced67-f8f8-480e-a1bc-4add3059fd0e","Type":"ContainerStarted","Data":"49a8d3afc1145860874a71203ef4f57b65a908c59bc122c41a7ced0f76455f98"} Nov 03 16:14:00.610101 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.610070 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" event={"ID":"45286eece609f171819d9469f4da982c","Type":"ContainerStarted","Data":"16e9d89b2d67cd85b7e2a67f80b1b9008d301b44f7f1353b64ee9c325c8a2364"} Nov 03 16:14:00.639234 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.639170 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" podStartSLOduration=6.63914861 podStartE2EDuration="6.63914861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:59.721765406 +0000 UTC m=+5.940850277" watchObservedRunningTime="2026-11-03 16:14:00.63914861 +0000 UTC m=+6.858233694" Nov 03 16:14:00.678763 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.678491 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h2qrh" podStartSLOduration=2.558799768 podStartE2EDuration="6.678473598s" totalImagesPullingTime="4.11967383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:00.639611502 +0000 UTC m=+6.858696378" watchObservedRunningTime="2026-11-03 16:14:00.678473598 +0000 UTC m=+6.897558473" Nov 03 16:14:00.980588 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.980348 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:14:00.981172 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:00.980963 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:14:01.020070 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:01.019602 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" podStartSLOduration=7.019588462 podStartE2EDuration="7.019588462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:00.678942409 +0000 UTC m=+6.898027280" watchObservedRunningTime="2026-11-03 16:14:01.019588462 +0000 UTC m=+7.238673335" Nov 03 16:14:01.425644 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:01.425607 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:01.425821 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:01.425748 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:01.426187 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:01.426164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:01.426300 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:01.426278 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:01.617623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:01.616791 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:14:01.617623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:01.617279 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:14:02.427725 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:02.427649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:02.427883 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:02.427787 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:02.623947 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:02.623905 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" event={"ID":"a8555d4b-9866-49fa-b635-d9b8e26f8b3b","Type":"ContainerStarted","Data":"e41301ddac86f28547c3a09cab106b17e5c78ab1935bf36aef7c4416ff4b9f85"} Nov 03 16:14:02.629950 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:02.629922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4tjbw" event={"ID":"a163d0db-7c08-45c9-bf83-6132d280a72a","Type":"ContainerStarted","Data":"f105ca5c34c0744feebf38f743319a6d99b3b917babe65ce1756989e7721dfdc"} Nov 03 16:14:03.078514 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.078410 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:03.078514 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.078498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:03.078760 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.078659 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:03.078760 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.078727 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:11.078707638 +0000 UTC m=+17.297792495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:03.079184 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.079163 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:03.079279 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.079222 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:11.079205783 +0000 UTC m=+17.298290637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:03.179247 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.179207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:03.179480 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.179419 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:14:03.179480 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.179463 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:14:03.179480 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.179477 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:03.179660 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.179541 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:14:11.179520569 +0000 UTC m=+17.398605437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:03.425742 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.425709 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:03.425883 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.425847 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:03.425952 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.425943 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:03.426055 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:03.426033 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:03.589531 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.584911 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:14:03.635220 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:03.634789 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" event={"ID":"a8555d4b-9866-49fa-b635-d9b8e26f8b3b","Type":"ContainerStarted","Data":"7dded3ae802ed62a512d9db5e194c57e1077031e5ff0f01a7ba6dd434a7f2168"} Nov 03 16:14:04.380309 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.379925 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:14:03.584932144Z","UUID":"b9818893-a65d-44b3-bea2-2c9562b95423","Handler":null,"Name":"","Endpoint":""} Nov 03 16:14:04.389716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.389677 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:14:04.389716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.389718 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:14:04.426532 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.426498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:04.426726 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:04.426627 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:04.641079 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.639681 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="efb691bd19bd4ab7c4bc80f295b258d9ed5eda0a29ec700cca8bfb24f4666f01" exitCode=0 Nov 03 16:14:04.641079 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.639789 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"efb691bd19bd4ab7c4bc80f295b258d9ed5eda0a29ec700cca8bfb24f4666f01"} Nov 03 16:14:04.642803 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.642540 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hwdlx" event={"ID":"fde7e318-0d30-4892-81f1-3e4f19c782f5","Type":"ContainerStarted","Data":"4137e02496b4269e8b0bcefe4c9facc19ed008cdd2961fe2e210b8da507600b3"} Nov 03 16:14:04.650740 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.649982 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t6s9c" event={"ID":"f6cd40ea-b486-4d7d-a8ec-49d153976a09","Type":"ContainerStarted","Data":"96233d0d9b103b46ddde2ba73b456aa789358c3110d35456029faef5e94226e9"} Nov 03 16:14:04.691214 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.691152 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4tjbw" podStartSLOduration=5.059388642 podStartE2EDuration="10.6911316s" totalImagesPullingTime="5.631742958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:02.671502205 +0000 UTC m=+8.890587080" watchObservedRunningTime="2026-11-03 16:14:04.6911316 +0000 UTC m=+10.910216474" Nov 03 16:14:04.779289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.779243 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-t6s9c" podStartSLOduration=2.789821175 podStartE2EDuration="10.77922777s" totalImagesPullingTime="7.989406595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:04.734803317 +0000 UTC m=+10.953888189" watchObservedRunningTime="2026-11-03 16:14:04.77922777 +0000 UTC m=+10.998312638" Nov 03 16:14:04.779425 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:04.779316 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hwdlx" podStartSLOduration=2.676411515 podStartE2EDuration="10.779312288s" totalImagesPullingTime="8.102900773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:04.779043052 +0000 UTC m=+10.998127924" watchObservedRunningTime="2026-11-03 16:14:04.779312288 +0000 UTC m=+10.998397200" Nov 03 16:14:05.425628 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.425546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:05.425628 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.425574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:05.425832 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:05.425665 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:05.425832 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:05.425804 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:05.654823 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.654786 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="2ee92c9301d17fbb20ae09ff9f99683d7f5902e0c91583b0e8f30b1f9a918c6b" exitCode=0 Nov 03 16:14:05.655387 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.654884 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"2ee92c9301d17fbb20ae09ff9f99683d7f5902e0c91583b0e8f30b1f9a918c6b"} Nov 03 16:14:05.658236 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.658211 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" event={"ID":"a8555d4b-9866-49fa-b635-d9b8e26f8b3b","Type":"ContainerStarted","Data":"a99e4d006d453bf3dfb2b170122334fe790b247b6e3deeb783c9454151b599c7"} Nov 03 16:14:05.660390 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.660359 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qb7s9" event={"ID":"0811b831-8592-4677-aa31-cd58433a3f5c","Type":"ContainerStarted","Data":"7344cc31e7743930515f3d4dc4a90ae98a9458229d20f5eb6a97cbc4fa8f8ea5"} Nov 03 16:14:05.746895 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.746793 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qb7s9" podStartSLOduration=3.371636699 podStartE2EDuration="11.746777739s" totalImagesPullingTime="8.37514104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:05.746466421 +0000 UTC m=+11.965551290" watchObservedRunningTime="2026-11-03 16:14:05.746777739 +0000 UTC m=+11.965862617" Nov 03 16:14:05.791965 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:05.791903 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" podStartSLOduration=3.242447704 podStartE2EDuration="11.791881719s" totalImagesPullingTime="8.549434015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:05.79177564 +0000 UTC m=+12.010860515" watchObservedRunningTime="2026-11-03 16:14:05.791881719 +0000 UTC m=+12.010966594" Nov 03 16:14:06.425854 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:06.425829 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:06.425966 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:06.425940 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:06.664198 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:06.664115 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lht4j" event={"ID":"746b87e4-de05-417f-82ba-53e481c27611","Type":"ContainerStarted","Data":"722424a6fd40d12d207108865d8671acea4233066b7b16ba389f9b9f8968c0e1"} Nov 03 16:14:06.667403 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:06.667373 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="ed44bcd3b27f12506a3f9d1591ad76333d0e979b872f65314bf70b74e71788bf" exitCode=0 Nov 03 16:14:06.667549 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:06.667474 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"ed44bcd3b27f12506a3f9d1591ad76333d0e979b872f65314bf70b74e71788bf"} Nov 03 16:14:06.708960 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:06.708907 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lht4j" podStartSLOduration=2.610651576 podStartE2EDuration="12.708892938s" totalImagesPullingTime="10.098241362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:06.708800876 +0000 UTC m=+12.927885749" watchObservedRunningTime="2026-11-03 16:14:06.708892938 +0000 UTC m=+12.927977810" Nov 03 16:14:07.425649 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:07.425609 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:07.425843 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:07.425737 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:07.425843 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:07.425815 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:07.425946 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:07.425890 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:08.426177 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:08.425979 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:08.426646 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:08.426270 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:09.425686 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:09.425649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:09.425871 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:09.425649 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:09.425871 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:09.425778 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:09.425986 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:09.425886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:09.677756 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:09.677670 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="a3bc1e4c80e3faed59a4789538f5ef1034e41ca5da26d8a9148764ae12f7b4c3" exitCode=0 Nov 03 16:14:09.677756 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:09.677734 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"a3bc1e4c80e3faed59a4789538f5ef1034e41ca5da26d8a9148764ae12f7b4c3"} Nov 03 16:14:10.426197 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.426174 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:10.426293 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:10.426277 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:10.683052 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.682882 2583 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="9c0d32e504812a831ea62ae73ce1e610c01c741c3c244abc785c7d7b70dc706e" exitCode=0 Nov 03 16:14:10.683476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.682955 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerDied","Data":"9c0d32e504812a831ea62ae73ce1e610c01c741c3c244abc785c7d7b70dc706e"} Nov 03 16:14:10.685271 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685255 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:14:10.685616 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685596 2583 generic.go:356] "Generic (PLEG): container finished" podID="d16e7255-2933-413f-9114-4b1e2f93d744" containerID="b454a445f22e12b81c1b6cdf4aacea05b5fa6811d9aca8f78f9e09357c41d9dd" exitCode=1 Nov 03 16:14:10.685684 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685629 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"e52559a14be9d76b8acc1dd2a449ebeeab25c8bc6c20a37edc6094e0b7ef5486"} Nov 03 16:14:10.685684 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685675 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"0cd743410c1a18bfd9c53462eb797764ade69f0a876cfb9dfbc3b6338303be5c"} Nov 03 16:14:10.685751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685706 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerDied","Data":"b454a445f22e12b81c1b6cdf4aacea05b5fa6811d9aca8f78f9e09357c41d9dd"} Nov 03 16:14:10.685751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:10.685735 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"0166061c1fff5843b3fbd79a4b479096341d1b2fc3e1f3b172c7e87513d2a630"} Nov 03 16:14:11.143857 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.143822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:11.143999 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.143866 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:11.143999 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.143975 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:11.144081 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.144033 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:27.144020997 +0000 UTC m=+33.363105861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:11.144081 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.143975 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:11.144146 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.144095 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:27.144083449 +0000 UTC m=+33.363168313 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:11.244930 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.244892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:11.245079 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.245024 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:14:11.245079 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.245040 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:14:11.245079 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.245051 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:11.245194 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.245101 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:14:27.245086535 +0000 UTC m=+33.464171396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:11.425717 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.425630 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:11.425855 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.425734 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:11.425855 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.425791 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:11.425914 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:11.425892 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:11.690377 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.690306 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:14:11.691221 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.691174 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"40ead1c6613fc4694f58de38dccd596bed79030a1c9b1727e4668717e939c9d6"} Nov 03 16:14:11.691361 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.691342 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"204d44b8d7390cd8525d878b11a22bfc0ea3f6afd5d81e0fd80746e5275c4184"} Nov 03 16:14:11.697096 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.697059 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" event={"ID":"0de08296-dbf4-45f3-8dc4-33cbf8de0658","Type":"ContainerStarted","Data":"59719b4c9175589adf2e778aecb8f7998e6d93a54385975c66774818193750cf"} Nov 03 16:14:11.771583 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:11.771536 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qdl9p" podStartSLOduration=7.371401008 podStartE2EDuration="17.771522654s" totalImagesPullingTime="10.400121646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:11.758760648 +0000 UTC m=+17.977845521" watchObservedRunningTime="2026-11-03 16:14:11.771522654 +0000 UTC m=+17.990607525" Nov 03 16:14:12.425719 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:12.425685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:12.425912 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:12.425793 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:13.425459 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:13.425408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:13.425823 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:13.425408 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:13.425823 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:13.425548 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:13.425823 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:13.425593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:13.705025 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:13.704961 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:14:13.705330 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:13.705309 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"9997b5276dd1cd9d7d043914b72e5290cad4fa9e48c20b87c5e88b31fcd694d1"} Nov 03 16:14:14.426307 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:14.426278 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:14.426686 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:14.426373 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:15.426052 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.426022 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:15.426188 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.426062 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:15.426188 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:15.426144 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:15.426264 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:15.426210 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:15.712250 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.712168 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:14:15.712901 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.712520 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"4374b10bc1f681f4179423a150f3ddf1c804e8ad32e9803c1ca055fe8a5095bd"} Nov 03 16:14:15.712955 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.712929 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:15.713043 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.713026 2583 scope.go:122] "RemoveContainer" containerID="b454a445f22e12b81c1b6cdf4aacea05b5fa6811d9aca8f78f9e09357c41d9dd" Nov 03 16:14:15.731060 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:15.731037 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:16.426176 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.426135 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:16.426347 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:16.426259 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:16.718328 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.718252 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:14:16.718836 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.718675 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" event={"ID":"d16e7255-2933-413f-9114-4b1e2f93d744","Type":"ContainerStarted","Data":"406f520bebe7e35781c9fdc1e7a448c2890da9d8f84969b3e6110dbec41dfea1"} Nov 03 16:14:16.718836 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.718773 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:16.719006 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.718991 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:16.734252 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.734228 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:16.814438 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:16.814377 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" podStartSLOduration=8.173625332 podStartE2EDuration="22.814363524s" totalImagesPullingTime="14.640738192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:16.800844611 +0000 UTC m=+23.019929483" watchObservedRunningTime="2026-11-03 16:14:16.814363524 +0000 UTC m=+23.033448396" Nov 03 16:14:17.425258 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:17.425230 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:17.425473 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:17.425232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:17.425473 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:17.425327 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:17.425473 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:17.425435 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:17.720988 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:17.720910 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:18.425629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:18.425593 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:18.425821 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:18.425709 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:18.723439 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:18.723362 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:19.332941 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:19.332915 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6s9c_f6cd40ea-b486-4d7d-a8ec-49d153976a09/dns-node-resolver/0.log" Nov 03 16:14:19.425810 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:19.425778 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:19.425970 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:19.425885 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:19.425970 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:19.425937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:19.426057 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:19.426040 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:20.064327 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:20.064297 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tjbw_a163d0db-7c08-45c9-bf83-6132d280a72a/node-ca/0.log" Nov 03 16:14:20.425654 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:20.425559 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:20.425811 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:20.425697 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:21.308954 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.308911 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:21.309488 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.309164 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:21.318877 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.318837 2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" podUID="d16e7255-2933-413f-9114-4b1e2f93d744" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 16:14:21.327351 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.327313 2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" podUID="d16e7255-2933-413f-9114-4b1e2f93d744" containerName="ovnkube-controller" probeResult="failure" output="" Nov 03 16:14:21.425532 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.425498 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:21.425532 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:21.425540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:21.425770 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:21.425645 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:21.425834 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:21.425785 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:22.425336 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:22.425294 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:22.425782 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:22.425435 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:23.425630 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:23.425592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:23.426086 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:23.425717 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:23.426086 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:23.425763 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:23.426086 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:23.425858 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:24.426044 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:24.426003 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:24.426636 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:24.426097 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:25.425567 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:25.425533 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:25.425763 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:25.425649 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:25.425763 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:25.425696 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:25.425843 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:25.425793 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:26.428964 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:26.428936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:26.429426 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:26.429034 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:27.150706 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:27.150665 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:27.150943 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:27.150715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:27.150943 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.150815 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:27.150943 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.150851 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:27.150943 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.150879 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret podName:a27eb936-8e3e-4bfb-8c6a-05c184ee89ea nodeName:}" failed. No retries permitted until 2026-11-03 16:14:59.150864496 +0000 UTC m=+65.369949351 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret") pod "global-pull-secret-syncer-4qqx2" (UID: "a27eb936-8e3e-4bfb-8c6a-05c184ee89ea") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:14:27.150943 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.150902 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs podName:bf95c1a4-4c19-4f20-8014-49afdffd700b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:59.1508873 +0000 UTC m=+65.369972165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs") pod "network-metrics-daemon-qgkpn" (UID: "bf95c1a4-4c19-4f20-8014-49afdffd700b") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:14:27.251719 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:27.251675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:27.251890 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.251840 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:14:27.251890 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.251858 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:14:27.251890 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.251868 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bj5mm for pod openshift-network-diagnostics/network-check-target-zhgz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:27.251991 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.251921 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm podName:b125f59a-0236-430f-9c41-57402f877adc nodeName:}" failed. No retries permitted until 2026-11-03 16:14:59.251905793 +0000 UTC m=+65.470990659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bj5mm" (UniqueName: "kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm") pod "network-check-target-zhgz9" (UID: "b125f59a-0236-430f-9c41-57402f877adc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:14:27.426173 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:27.426079 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:27.426355 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:27.426081 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:27.426355 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.426221 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:27.426355 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:27.426272 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:28.426226 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:28.426190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:28.426660 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:28.426299 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:29.426182 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:29.426143 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:29.426402 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:29.426157 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:29.426402 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:29.426287 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:29.426402 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:29.426374 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:30.425727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:30.425698 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:30.425899 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:30.425803 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:31.425256 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:31.425223 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:31.425657 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:31.425228 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:31.425657 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:31.425337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:31.425657 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:31.425408 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:32.425485 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:32.425429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:32.425987 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:32.425568 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:33.425870 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:33.425830 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:33.425870 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:33.425855 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:33.426400 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:33.425976 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:33.426400 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:33.426078 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:34.426092 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:34.426059 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:34.426511 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:34.426145 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:35.425576 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:35.425529 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:35.425576 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:35.425557 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:35.425843 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:35.425650 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:35.425843 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:35.425783 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:36.076199 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.075878 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qqx2"] Nov 03 16:14:36.076719 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.076322 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:36.076719 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:36.076501 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:36.076719 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.076591 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhgz9"] Nov 03 16:14:36.076719 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.076671 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:36.076933 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:36.076747 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:36.077410 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.077373 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgkpn"] Nov 03 16:14:36.077567 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:36.077493 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:36.077629 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:36.077599 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:37.426363 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:37.426262 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:37.426919 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:37.426282 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:37.426919 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:37.426386 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:37.426919 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:37.426288 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:37.426919 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:37.426482 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:37.426919 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:37.426576 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:39.426333 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:39.426246 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:39.426333 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:39.426269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:39.426829 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:39.426412 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhgz9" podUID="b125f59a-0236-430f-9c41-57402f877adc" Nov 03 16:14:39.426829 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:39.426429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:39.426829 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:39.426504 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4qqx2" podUID="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" Nov 03 16:14:39.426829 ip-10-0-129-130 kubenswrapper[2583]: E1103 16:14:39.426571 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qgkpn" podUID="bf95c1a4-4c19-4f20-8014-49afdffd700b" Nov 03 16:14:40.577095 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.577060 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeReady" Nov 03 16:14:40.577616 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.577211 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 16:14:40.646625 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.646584 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2dsmp"] Nov 03 16:14:40.648478 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.648434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.650379 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.650357 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rnjkc"] Nov 03 16:14:40.651802 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.651783 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.673746 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.673722 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:14:40.675121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.675106 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:14:40.675166 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.675142 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:14:40.685428 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.685407 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2dsmp"] Nov 03 16:14:40.686121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.686102 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnjkc"] Nov 03 16:14:40.686870 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.686850 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-phsml"] Nov 03 16:14:40.689068 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.689051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.749899 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.749858 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.749899 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.749898 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2sdp\" (UniqueName: \"kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.750144 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.750006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.750144 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.750036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.750144 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.750134 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.750253 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.750159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6928\" (UniqueName: \"kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.765653 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.765624 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phsml"] Nov 03 16:14:40.768836 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.768805 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:14:40.770681 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.770663 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.780807 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.780787 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:14:40.781013 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.780998 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:14:40.781082 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.781036 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:14:40.781082 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.781054 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:14:40.781273 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.781260 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:14:40.788706 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.788685 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:14:40.788833 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.788731 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:14:40.788833 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.788738 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:14:40.788833 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.788773 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:14:40.792957 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.792936 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 16:14:40.793335 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.793322 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 16:14:40.793458 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.793436 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 16:14:40.793512 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.793461 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 16:14:40.793552 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.793523 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 16:14:40.797820 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.797802 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:14:40.851299 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.851299 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851253 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.851299 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.851299 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.851600 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851361 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.851600 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.851600 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sdp\" (UniqueName: \"kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.851986 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852171 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.852370 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzt6k\" (UniqueName: \"kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.852840 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.852959 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.852959 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.852940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.853058 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.853007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6928\" (UniqueName: \"kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.853104 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.853055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.853104 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.853082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.855704 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.853326 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.858568 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.858532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.858663 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.858595 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.874279 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.874216 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 16:14:40.874462 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.874297 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qrl28\"" Nov 03 16:14:40.874462 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.874302 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 16:14:40.876195 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.876169 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sdp\" (UniqueName: \"kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.876689 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.876671 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6928\" (UniqueName: \"kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.953647 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953604 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.953647 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.953928 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953664 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.953928 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.953928 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.953928 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953915 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzt6k\" (UniqueName: \"kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.954126 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.953945 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.954126 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.954240 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954220 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.954289 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.954344 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954288 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.954344 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.954646 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.954616 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.955148 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955120 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.955323 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955178 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.955323 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955192 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.955323 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955207 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.955507 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955361 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.955665 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.955642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.956386 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.956371 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.956486 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.956462 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:40.957011 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.956992 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.961994 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.961975 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:40.965349 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.965334 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:14:40.978934 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.978889 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:40.981858 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:40.981833 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzt6k\" (UniqueName: \"kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:41.000218 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.000182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:14:41.080872 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.080834 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:41.281756 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.281732 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rnjkc"] Nov 03 16:14:41.285493 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.285419 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2dsmp"] Nov 03 16:14:41.292403 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:41.292378 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26796971_9faf_4381_8f8b_23db59a37fcc.slice/crio-f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b WatchSource:0}: Error finding container f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b: Status 404 returned error can't find the container with id f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b Nov 03 16:14:41.310958 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.310932 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phsml"] Nov 03 16:14:41.311495 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:41.311468 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f18098b_6d38_4cba_85ed_5e110a988df8.slice/crio-591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6 WatchSource:0}: Error finding container 591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6: Status 404 returned error can't find the container with id 591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6 Nov 03 16:14:41.376512 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.376439 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:14:41.377054 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:41.377016 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e84b0c_2265_41de_ad75_1d9329e1dc21.slice/crio-1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf WatchSource:0}: Error finding container 1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf: Status 404 returned error can't find the container with id 1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf Nov 03 16:14:41.426064 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.426023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:41.426265 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.426091 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:41.426265 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.426097 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:41.441542 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.441513 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:41.441648 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.441610 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:41.441934 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.441911 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:41.442033 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.441948 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:41.442033 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.441990 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:41.442127 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.442055 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:41.442127 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.442107 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:41.783014 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.782941 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnjkc" event={"ID":"e757f958-921a-4cc9-9735-c3f5594b55d9","Type":"ContainerStarted","Data":"d516a932e9ab0922cdcd8448881e85e124db634da2d74adec15777400d30e7ab"} Nov 03 16:14:41.785615 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.785584 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7765985f69-vdrnv" event={"ID":"70e84b0c-2265-41de-ad75-1d9329e1dc21","Type":"ContainerStarted","Data":"1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf"} Nov 03 16:14:41.787825 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.787715 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dsmp" event={"ID":"26796971-9faf-4381-8f8b-23db59a37fcc","Type":"ContainerStarted","Data":"f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b"} Nov 03 16:14:41.790537 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.790509 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phsml" event={"ID":"4f18098b-6d38-4cba-85ed-5e110a988df8","Type":"ContainerStarted","Data":"e32347624e2ad019804914e706c43d81a96684f37ea90c058f4215599ea9e6eb"} Nov 03 16:14:41.790641 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:41.790576 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phsml" event={"ID":"4f18098b-6d38-4cba-85ed-5e110a988df8","Type":"ContainerStarted","Data":"591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6"} Nov 03 16:14:42.797579 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:42.797540 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phsml" event={"ID":"4f18098b-6d38-4cba-85ed-5e110a988df8","Type":"ContainerStarted","Data":"29a2cddbf05e1715d67669e171c4c04acf07ef3ee33c4f22b29faba1fd31f93b"} Nov 03 16:14:43.807519 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:43.807479 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dsmp" event={"ID":"26796971-9faf-4381-8f8b-23db59a37fcc","Type":"ContainerStarted","Data":"b988d7d3f4809dbb6cd23358eb8a5b90ee62cba9b88de561b8246e24cd65f080"} Nov 03 16:14:43.807907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:43.807587 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2dsmp" event={"ID":"26796971-9faf-4381-8f8b-23db59a37fcc","Type":"ContainerStarted","Data":"564e04633a1743d2242cc74f9bd24f7547b6dea013348c0b3229ddefd82f8d1c"} Nov 03 16:14:43.807907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:43.807752 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:43.852340 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:43.852278 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-2dsmp" podStartSLOduration=1.7416590539999999 podStartE2EDuration="3.852257136s" totalImagesPullingTime="2.110598082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:43.85099024 +0000 UTC m=+50.070075114" watchObservedRunningTime="2026-11-03 16:14:43.852257136 +0000 UTC m=+50.071342010" Nov 03 16:14:44.444669 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.444597 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz"] Nov 03 16:14:44.446478 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.446436 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.468727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468531 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:14:44.468727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468615 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:14:44.468727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468703 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-drxjc\"" Nov 03 16:14:44.468727 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468727 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:14:44.469026 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468779 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:14:44.469026 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.468785 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:14:44.482350 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.482319 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz"] Nov 03 16:14:44.497773 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.497745 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n9q8w"] Nov 03 16:14:44.500530 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.500510 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.514025 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.513999 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7"] Nov 03 16:14:44.516161 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.516142 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.539884 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.539858 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7"] Nov 03 16:14:44.575389 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.575362 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 16:14:44.576589 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.576565 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:14:44.576916 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.576880 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 16:14:44.577042 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.577027 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cpmbj\"" Nov 03 16:14:44.577219 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.577129 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p9flk\"" Nov 03 16:14:44.577219 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.577153 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:14:44.577311 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.577244 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:14:44.577610 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.577575 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.585879 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.585906 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.585923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjntj\" (UniqueName: \"kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.585962 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.585995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586031 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586072 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvp5\" (UniqueName: \"kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.586362 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.586186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687598 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687555 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjntj\" (UniqueName: \"kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687766 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687794 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.687848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvp5\" (UniqueName: \"kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687946 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rzx\" (UniqueName: \"kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.687974 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688057 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688086 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688118 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.688183 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.688636 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688254 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.688934 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.688905 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.689623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.689599 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.689760 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.689743 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.691920 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.691273 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.691920 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.691501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.691920 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.691850 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.693133 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.693087 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.694647 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.694277 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.694647 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.694608 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.696471 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.696377 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.716112 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.716073 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjntj\" (UniqueName: \"kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.716252 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.716147 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvp5\" (UniqueName: \"kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.761483 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.761434 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:14:44.788879 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.788849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789017 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.788936 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789017 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.788985 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rzx\" (UniqueName: \"kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789122 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789122 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789122 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789097 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789653 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789617 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789784 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789768 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.789880 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.789857 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.791801 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.791778 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.791869 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.791856 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.814626 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.814574 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phsml" event={"ID":"4f18098b-6d38-4cba-85ed-5e110a988df8","Type":"ContainerStarted","Data":"dc87507d034587566814cbf3ba2b7c36d5048660a53b89c29ef1fd254be29fa7"} Nov 03 16:14:44.817544 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.815798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rzx\" (UniqueName: \"kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:44.817544 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.817186 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rnjkc" event={"ID":"e757f958-921a-4cc9-9735-c3f5594b55d9","Type":"ContainerStarted","Data":"6e2619b300e1d6494ac218c10017a9cb29e80e74f739f2f93de7aa28649e691f"} Nov 03 16:14:44.820234 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.820042 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:14:44.835146 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:44.835121 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa2b183_e4e8_41fc_85e4_52293e0add64.slice/crio-d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04 WatchSource:0}: Error finding container d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04: Status 404 returned error can't find the container with id d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04 Nov 03 16:14:44.835467 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:44.835368 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:14:45.025916 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.025302 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-phsml" podStartSLOduration=2.708949808 podStartE2EDuration="5.025284564s" totalImagesPullingTime="2.316334756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:14:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:44.974338313 +0000 UTC m=+51.193423185" watchObservedRunningTime="2026-11-03 16:14:45.025284564 +0000 UTC m=+51.244369438" Nov 03 16:14:45.060349 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.060318 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz"] Nov 03 16:14:45.060898 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:45.060870 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13a995e_bff6_4420_b8d3_4d2f107c2fa4.slice/crio-31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e WatchSource:0}: Error finding container 31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e: Status 404 returned error can't find the container with id 31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e Nov 03 16:14:45.158312 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.158266 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rnjkc" podStartSLOduration=1.9716385619999999 podStartE2EDuration="5.1582507s" totalImagesPullingTime="3.186612138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:45.15757541 +0000 UTC m=+51.376660283" watchObservedRunningTime="2026-11-03 16:14:45.1582507 +0000 UTC m=+51.377335572" Nov 03 16:14:45.180716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.180689 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7"] Nov 03 16:14:45.182400 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:45.182371 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeeed625_5069_4da6_8482_2c8ca7c4ff88.slice/crio-a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790 WatchSource:0}: Error finding container a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790: Status 404 returned error can't find the container with id a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790 Nov 03 16:14:45.829408 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.828725 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7765985f69-vdrnv" event={"ID":"70e84b0c-2265-41de-ad75-1d9329e1dc21","Type":"ContainerStarted","Data":"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc"} Nov 03 16:14:45.830452 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.830409 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" event={"ID":"deeed625-5069-4da6-8482-2c8ca7c4ff88","Type":"ContainerStarted","Data":"a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790"} Nov 03 16:14:45.831992 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.831962 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9q8w" event={"ID":"caa2b183-e4e8-41fc-85e4-52293e0add64","Type":"ContainerStarted","Data":"d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04"} Nov 03 16:14:45.835663 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.835637 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" event={"ID":"a13a995e-bff6-4420-b8d3-4d2f107c2fa4","Type":"ContainerStarted","Data":"2d91090e43a2eb5c4095caca0819e54044bfa6a3bd25b35aa24d7225a08b3c49"} Nov 03 16:14:45.835781 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.835694 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" event={"ID":"a13a995e-bff6-4420-b8d3-4d2f107c2fa4","Type":"ContainerStarted","Data":"37b3d7e73da9edfb634da5830d1976d6126834ad4fe554b63ff386586681b39d"} Nov 03 16:14:45.835781 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.835713 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" event={"ID":"a13a995e-bff6-4420-b8d3-4d2f107c2fa4","Type":"ContainerStarted","Data":"31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e"} Nov 03 16:14:45.848058 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.848034 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:14:45.850426 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.850401 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:45.875103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.870170 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:14:45.875103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.870214 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4j92k\"" Nov 03 16:14:45.875103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.874123 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:14:45.880639 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.880111 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7765985f69-vdrnv" podStartSLOduration=2.454395613 podStartE2EDuration="5.880099851s" totalImagesPullingTime="3.425704238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:45.879461755 +0000 UTC m=+52.098546628" watchObservedRunningTime="2026-11-03 16:14:45.880099851 +0000 UTC m=+52.099184722" Nov 03 16:14:45.884634 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.884569 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:14:45.900210 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.900183 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:14:45.971547 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971159 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:14:45.971547 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971353 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:14:45.971547 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971426 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:14:45.971547 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971482 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:14:45.971547 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971548 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:14:45.971846 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:45.971627 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:14:46.000434 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000386 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000434 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000466 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000586 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000613 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000639 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000697 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000929 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000929 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000831 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.000929 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000885 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.001062 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.001062 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.000996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.001062 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.001039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103754 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103754 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103710 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103763 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103852 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103872 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103902 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.103967 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103950 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.104288 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.103983 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.104288 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.104044 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.104288 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.104082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.104288 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.104110 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.105200 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.104959 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.105862 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.105834 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.106037 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.106008 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.108287 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.108246 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.108688 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.108662 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.109321 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.109295 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.110605 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.110563 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.110689 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.110612 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.111462 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.111428 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.111560 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.111475 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.111863 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.111838 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.115010 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.114991 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.135118 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.135069 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.171956 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.171928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:14:46.506954 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.506922 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:14:46.519514 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:46.519486 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92928fd5_df80_494a_b99f_1e675a190dea.slice/crio-685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca WatchSource:0}: Error finding container 685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca: Status 404 returned error can't find the container with id 685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca Nov 03 16:14:46.661540 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.661462 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5765448457-qf2dp"] Nov 03 16:14:46.663899 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.663875 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.682332 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.682305 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ccbfl\"" Nov 03 16:14:46.682556 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.682501 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ddc504v5k29s\"" Nov 03 16:14:46.682882 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.682862 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 16:14:46.683012 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.682997 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 16:14:46.683097 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.683080 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 16:14:46.683097 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.681622 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 16:14:46.683351 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.683335 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 16:14:46.692657 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.692635 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5765448457-qf2dp"] Nov 03 16:14:46.810292 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810258 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810292 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810563 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810325 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87cd7\" (UniqueName: \"kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810563 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810425 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810563 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810563 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810516 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.810751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.810719 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.840411 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.840376 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca"} Nov 03 16:14:46.842633 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.842597 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" event={"ID":"deeed625-5069-4da6-8482-2c8ca7c4ff88","Type":"ContainerStarted","Data":"bc9f26b9bdabfd1952756b3610b678e9be402afe3d11af51bc45d5ec9155a7ff"} Nov 03 16:14:46.842770 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.842651 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" event={"ID":"deeed625-5069-4da6-8482-2c8ca7c4ff88","Type":"ContainerStarted","Data":"d0c3010ac11e4d3d69a232bc2992cd0216b9122e90112c58827c2d577d144c21"} Nov 03 16:14:46.842770 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.842681 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" event={"ID":"deeed625-5069-4da6-8482-2c8ca7c4ff88","Type":"ContainerStarted","Data":"179fbc64f2990795f934f5ec0cda2a0dac292268c0467d4dd7fd7a93b0924cf7"} Nov 03 16:14:46.844265 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.844237 2583 generic.go:356] "Generic (PLEG): container finished" podID="caa2b183-e4e8-41fc-85e4-52293e0add64" containerID="b3adc34df20f0a7839a2e9ac3c0d631ee7da38b40b54f741357176b0e9f254e5" exitCode=0 Nov 03 16:14:46.844402 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.844312 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9q8w" event={"ID":"caa2b183-e4e8-41fc-85e4-52293e0add64","Type":"ContainerDied","Data":"b3adc34df20f0a7839a2e9ac3c0d631ee7da38b40b54f741357176b0e9f254e5"} Nov 03 16:14:46.846613 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.846585 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" event={"ID":"a13a995e-bff6-4420-b8d3-4d2f107c2fa4","Type":"ContainerStarted","Data":"0a1d5909cf032302f4703169d6fd54afbec2aaa1180e9a4463f5f8c12b63e2a8"} Nov 03 16:14:46.911716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cd7\" (UniqueName: \"kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911716 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911707 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911811 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911830 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911860 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.911945 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.911893 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.913876 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.913839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.913998 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.913938 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.915017 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.914966 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.916169 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.916128 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.917738 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.917616 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.918332 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.918309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.919575 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.919518 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.921877 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.921851 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.941581 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.941534 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" podStartSLOduration=1.735926739 podStartE2EDuration="2.941517582s" totalImagesPullingTime="1.205590843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:46.890394549 +0000 UTC m=+53.109479421" watchObservedRunningTime="2026-11-03 16:14:46.941517582 +0000 UTC m=+53.160602456" Nov 03 16:14:46.946479 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.946437 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cd7\" (UniqueName: \"kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.988401 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.988377 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:46.999103 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:46.994274 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" podStartSLOduration=1.978178866 podStartE2EDuration="2.994255928s" totalImagesPullingTime="1.016077062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:46.993097583 +0000 UTC m=+53.212182457" watchObservedRunningTime="2026-11-03 16:14:46.994255928 +0000 UTC m=+53.213340801" Nov 03 16:14:47.298157 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.298131 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5765448457-qf2dp"] Nov 03 16:14:47.298773 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:47.298741 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdfec9a_b17a_4fc9_bcb1_2b0be3a97c54.slice/crio-e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40 WatchSource:0}: Error finding container e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40: Status 404 returned error can't find the container with id e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40 Nov 03 16:14:47.853524 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.853483 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9q8w" event={"ID":"caa2b183-e4e8-41fc-85e4-52293e0add64","Type":"ContainerStarted","Data":"7fd5ed00a7d244d418c6e0891ed8d570bf3da533a1c8e7d96a04afd571bfa618"} Nov 03 16:14:47.854112 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.853554 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n9q8w" event={"ID":"caa2b183-e4e8-41fc-85e4-52293e0add64","Type":"ContainerStarted","Data":"d0624a17a8e691670aac83edfe29be2313e6098d127d0b47736ce9925d656409"} Nov 03 16:14:47.855573 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.855548 2583 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" exitCode=0 Nov 03 16:14:47.855706 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.855648 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8"} Nov 03 16:14:47.856987 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.856963 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40"} Nov 03 16:14:47.901456 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:47.901390 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n9q8w" podStartSLOduration=2.8131721389999997 podStartE2EDuration="3.901376665s" totalImagesPullingTime="1.018918186s" totalInitContainerRuntime="69.28634ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:47.899619425 +0000 UTC m=+54.118704299" watchObservedRunningTime="2026-11-03 16:14:47.901376665 +0000 UTC m=+54.120461538" Nov 03 16:14:49.109192 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.109149 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:14:49.111621 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.111596 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.167050 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.166997 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 16:14:49.169239 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.169143 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:14:49.172468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.172430 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68fbd467cc-f5zvj"] Nov 03 16:14:49.176786 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.176762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.199411 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.199368 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 16:14:49.199579 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.199417 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e8gqemq6h18bl\"" Nov 03 16:14:49.199579 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.199562 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 16:14:49.199909 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.199877 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k6446\"" Nov 03 16:14:49.201518 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.201494 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68fbd467cc-f5zvj"] Nov 03 16:14:49.227218 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.227187 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 16:14:49.227332 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.227318 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 16:14:49.234553 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234519 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.234656 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234598 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.234656 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.234765 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234678 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.234905 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.235016 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.234921 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.235086 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.235036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.235144 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.235090 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.259749 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.259723 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n97fl"] Nov 03 16:14:49.263654 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.263638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.278924 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.278900 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n97fl"] Nov 03 16:14:49.303524 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.303500 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5zgcl\"" Nov 03 16:14:49.303666 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.303599 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336553 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336705 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336736 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336801 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336917 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.337429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.336974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.339283 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.338831 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.339607 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.339381 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.339607 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.339501 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.339607 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.339510 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.340037 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.340001 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.340100 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.340064 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.340913 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.340882 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.363617 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.363548 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.423219 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.423182 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437735 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437789 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437806 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437827 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7gg\" (UniqueName: \"kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.437920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.438468 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.438332 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.439066 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.439044 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.439905 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.439878 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.440568 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.440546 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.441150 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.441127 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.441832 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.441812 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.462181 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.462148 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.501850 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.501815 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:14:49.538538 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.538502 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.538699 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.538548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7gg\" (UniqueName: \"kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.541479 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.541429 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.565748 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.565711 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7gg\" (UniqueName: \"kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.576741 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.576708 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:49.724104 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.724078 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:14:49.724400 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:49.724379 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a16c09_5008_44a0_b655_7b1f12ab2317.slice/crio-437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a WatchSource:0}: Error finding container 437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a: Status 404 returned error can't find the container with id 437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a Nov 03 16:14:49.800650 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.800305 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68fbd467cc-f5zvj"] Nov 03 16:14:49.804580 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:49.804546 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefef1ee8_f44f_4b0f_9c5d_de4024cfb8da.slice/crio-b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652 WatchSource:0}: Error finding container b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652: Status 404 returned error can't find the container with id b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652 Nov 03 16:14:49.866977 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.866888 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" event={"ID":"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da","Type":"ContainerStarted","Data":"b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652"} Nov 03 16:14:49.870907 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.870743 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea"} Nov 03 16:14:49.871032 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.870944 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e"} Nov 03 16:14:49.871032 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.870987 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0"} Nov 03 16:14:49.876123 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.875936 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"072e16a32bac54e19ffc3907df2e802b96da5a78adb345a932983d9c1513fbe0"} Nov 03 16:14:49.876123 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.876012 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"376fa90a88d5b81647af93ff96417dfcb44e4e325f6bb02e51c415e55740a0e2"} Nov 03 16:14:49.876123 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.876052 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"1e15d7fb9a66a470045c6def83e91cd6414ce34ab38d39c39c00a6cd3ee2d423"} Nov 03 16:14:49.879686 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.878849 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597cf7cc48-9wxwj" event={"ID":"68a16c09-5008-44a0-b655-7b1f12ab2317","Type":"ContainerStarted","Data":"3d3a67e21520bbb81e2bf85495575cba013705679c748a2c33f1254998cf74f8"} Nov 03 16:14:49.879686 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.878895 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597cf7cc48-9wxwj" event={"ID":"68a16c09-5008-44a0-b655-7b1f12ab2317","Type":"ContainerStarted","Data":"437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a"} Nov 03 16:14:49.887552 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.887263 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-n97fl"] Nov 03 16:14:49.888360 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:49.888281 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc21144a_4ab2_4b38_b190_54a63a17c90c.slice/crio-c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89 WatchSource:0}: Error finding container c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89: Status 404 returned error can't find the container with id c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89 Nov 03 16:14:49.929109 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:49.929048 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-597cf7cc48-9wxwj" podStartSLOduration=0.929029726 podStartE2EDuration="929.029726ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:49.928546692 +0000 UTC m=+56.147631566" watchObservedRunningTime="2026-11-03 16:14:49.929029726 +0000 UTC m=+56.148114595" Nov 03 16:14:50.888640 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.888599 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8"} Nov 03 16:14:50.889125 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.888681 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0"} Nov 03 16:14:50.889125 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.888709 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerStarted","Data":"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91"} Nov 03 16:14:50.892783 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.892034 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"328c790bd5f5fa4a0e5bad41e5f569ff1bc811bfff6d106af9a55de58fa5b323"} Nov 03 16:14:50.892783 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.892108 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"db7b7deee4dd2aa3566a36635f48f6e2f5d6ef9e7e503e225a36108cd89f99a4"} Nov 03 16:14:50.892783 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.892158 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" event={"ID":"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54","Type":"ContainerStarted","Data":"85e4126a7c007c0623d68aaf6782a7ea76232560e948e77c29b356e5c7cdf3c4"} Nov 03 16:14:50.892783 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.892205 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:50.893263 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.893234 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" event={"ID":"bc21144a-4ab2-4b38-b190-54a63a17c90c","Type":"ContainerStarted","Data":"c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89"} Nov 03 16:14:50.992185 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:50.992112 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.052109812 podStartE2EDuration="5.992095535s" totalImagesPullingTime="2.924241132s" totalInitContainerRuntime="15.744591ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:50.991977969 +0000 UTC m=+57.211063082" watchObservedRunningTime="2026-11-03 16:14:50.992095535 +0000 UTC m=+57.211180408" Nov 03 16:14:51.052594 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.052541 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" podStartSLOduration=2.438465718 podStartE2EDuration="5.052523572s" totalImagesPullingTime="2.614057854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:51.050359336 +0000 UTC m=+57.269444210" watchObservedRunningTime="2026-11-03 16:14:51.052523572 +0000 UTC m=+57.271608445" Nov 03 16:14:51.082476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.081708 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:51.082476 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.081778 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:51.087553 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.087529 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:51.330527 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.330424 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:14:51.400509 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.400474 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:14:51.404323 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.404304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.429017 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.428989 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 16:14:51.429239 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.429226 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 16:14:51.429302 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.429253 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 16:14:51.443248 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.443217 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 16:14:51.443739 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.443709 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 16:14:51.456301 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.456274 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:14:51.460882 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.460855 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 16:14:51.461233 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461212 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 16:14:51.461348 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461330 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ehkrv9jhbpoh\"" Nov 03 16:14:51.461409 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461372 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kx5mg\"" Nov 03 16:14:51.461409 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461370 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 16:14:51.461409 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461397 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 16:14:51.461561 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461465 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 16:14:51.461561 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461496 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 16:14:51.461561 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461508 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 16:14:51.461561 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461515 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 16:14:51.461669 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461609 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461669 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461669 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461667 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461764 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461849 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461820 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461849 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461836 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 16:14:51.461980 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461864 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461980 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461980 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.461980 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.461994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462039 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462074 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462102 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462121 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462322 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462197 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462322 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462257 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462322 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462295 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462323 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462362 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.462474 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.462422 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563141 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563141 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563111 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563141 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563140 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563169 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563335 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563404 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563392 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563481 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563512 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563561 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563607 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.563794 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563773 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.564221 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563999 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.564279 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.564239 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.564458 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.563533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.564761 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.564736 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.564995 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.564959 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.566416 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.566104 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.566885 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.566864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.567623 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.567553 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.567722 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.567671 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.567781 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.567728 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568302 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568270 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568381 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568277 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568429 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568381 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568689 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568897 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568873 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.568977 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.568948 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.569125 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.569106 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.569828 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.569808 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.570038 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.570015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.590903 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.590877 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.731748 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.731685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:14:51.899174 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.899088 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" event={"ID":"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da","Type":"ContainerStarted","Data":"3a551ca38ba282b0caaf0c862c9a6b952a1c9263e73e60d80362e85bf068bdf8"} Nov 03 16:14:51.901099 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.901065 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" event={"ID":"bc21144a-4ab2-4b38-b190-54a63a17c90c","Type":"ContainerStarted","Data":"8fa0cb9bb32e82e733dd78964d66779767d38177b3bdc76d74cdb693d80744b8"} Nov 03 16:14:51.901824 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.901802 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:51.906100 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.906079 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:14:51.907200 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.907179 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:14:51.946994 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:51.946945 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" podStartSLOduration=1.4946931079999999 podStartE2EDuration="2.946927125s" totalImagesPullingTime="1.452234017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:51.945591452 +0000 UTC m=+58.164676339" watchObservedRunningTime="2026-11-03 16:14:51.946927125 +0000 UTC m=+58.166011996" Nov 03 16:14:52.033285 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:52.033233 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" podStartSLOduration=1.6620692259999998 podStartE2EDuration="3.033217254s" totalImagesPullingTime="1.371148028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:52.032281213 +0000 UTC m=+58.251366067" watchObservedRunningTime="2026-11-03 16:14:52.033217254 +0000 UTC m=+58.252302127" Nov 03 16:14:52.068942 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:52.068906 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:14:52.076766 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:52.076731 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391a45b7_3643_4fa0_994f_dc590a257672.slice/crio-9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6 WatchSource:0}: Error finding container 9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6: Status 404 returned error can't find the container with id 9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6 Nov 03 16:14:52.906401 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:52.906368 2583 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496" exitCode=0 Nov 03 16:14:52.906876 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:52.906462 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496"} Nov 03 16:14:52.906876 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:52.906549 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6"} Nov 03 16:14:53.819602 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:53.819572 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2dsmp" Nov 03 16:14:56.762419 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.762389 2583 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:14:56.852863 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.852821 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:14:56.856283 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.856257 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.889710 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.889678 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:14:56.907072 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.907045 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:14:56.915848 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.915822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.915988 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.915875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916085 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.915997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916085 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.916045 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916228 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.916103 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916288 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.916270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916509 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.916360 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.916509 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.916411 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:56.924305 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924277 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c"} Nov 03 16:14:56.924406 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924361 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93"} Nov 03 16:14:56.924406 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924386 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89"} Nov 03 16:14:56.924502 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924411 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4"} Nov 03 16:14:56.924502 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924437 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76"} Nov 03 16:14:56.924502 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:56.924483 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerStarted","Data":"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f"} Nov 03 16:14:57.017809 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.017726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.017969 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.017927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.018029 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.017981 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.018139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.018769 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.018907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.019022 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.019230 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.019372 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.019556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020071 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.019647 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020579 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.020127 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.020579 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.020479 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.021113 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.021089 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.021569 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.021543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.040974 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.040925 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.897971537 podStartE2EDuration="6.04090762s" totalImagesPullingTime="3.121705268s" totalInitContainerRuntime="21.230815ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:57.039831694 +0000 UTC m=+63.258916566" watchObservedRunningTime="2026-11-03 16:14:57.04090762 +0000 UTC m=+63.259992493" Nov 03 16:14:57.048997 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.048976 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.167952 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.167915 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:14:57.469821 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.469789 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:14:57.469974 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:57.469951 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46511848_529c_41d9_a9ae_d09fa48ddd3d.slice/crio-8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e WatchSource:0}: Error finding container 8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e: Status 404 returned error can't find the container with id 8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e Nov 03 16:14:57.930076 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.930038 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7b79479-75lgp" event={"ID":"46511848-529c-41d9-a9ae-d09fa48ddd3d","Type":"ContainerStarted","Data":"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432"} Nov 03 16:14:57.930546 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.930089 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7b79479-75lgp" event={"ID":"46511848-529c-41d9-a9ae-d09fa48ddd3d","Type":"ContainerStarted","Data":"8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e"} Nov 03 16:14:57.979454 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:57.979402 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54d7b79479-75lgp" podStartSLOduration=1.9793885869999999 podStartE2EDuration="1.979388587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:57.977227629 +0000 UTC m=+64.196312500" watchObservedRunningTime="2026-11-03 16:14:57.979388587 +0000 UTC m=+64.198473458" Nov 03 16:14:59.249435 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.249396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:59.249870 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.249463 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:59.264629 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.264606 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:59.271792 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.271762 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:59.291501 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.291475 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:59.302668 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.302642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:59.350374 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.350343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:59.365032 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.365001 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:59.385810 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.385788 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:59.393751 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.393723 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:59.423815 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.423785 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:14:59.452615 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.452583 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:59.457016 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.456996 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:59.459102 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.459086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:14:59.462475 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.462458 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:59.462786 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.462769 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:14:59.468523 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.468501 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:14:59.760952 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.760928 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4qqx2"] Nov 03 16:14:59.762656 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:59.762630 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27eb936_8e3e_4bfb_8c6a_05c184ee89ea.slice/crio-e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05 WatchSource:0}: Error finding container e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05: Status 404 returned error can't find the container with id e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05 Nov 03 16:14:59.775029 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.775003 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qgkpn"] Nov 03 16:14:59.776047 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:59.776023 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf95c1a4_4c19_4f20_8014_49afdffd700b.slice/crio-cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86 WatchSource:0}: Error finding container cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86: Status 404 returned error can't find the container with id cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86 Nov 03 16:14:59.779869 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.779849 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhgz9"] Nov 03 16:14:59.780061 ip-10-0-129-130 kubenswrapper[2583]: W1103 16:14:59.780042 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb125f59a_0236_430f_9c41_57402f877adc.slice/crio-25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c WatchSource:0}: Error finding container 25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c: Status 404 returned error can't find the container with id 25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c Nov 03 16:14:59.940615 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.940582 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4qqx2" event={"ID":"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea","Type":"ContainerStarted","Data":"e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05"} Nov 03 16:14:59.941713 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.941690 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhgz9" event={"ID":"b125f59a-0236-430f-9c41-57402f877adc","Type":"ContainerStarted","Data":"25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c"} Nov 03 16:14:59.942824 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:14:59.942803 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgkpn" event={"ID":"bf95c1a4-4c19-4f20-8014-49afdffd700b","Type":"ContainerStarted","Data":"cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86"} Nov 03 16:15:01.732666 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:01.732623 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:01.963141 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:01.963095 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgkpn" event={"ID":"bf95c1a4-4c19-4f20-8014-49afdffd700b","Type":"ContainerStarted","Data":"e6c02f42bab79c7353f20905d41f5689c037f377f9cc3d7152d2fa34217868d5"} Nov 03 16:15:01.963329 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:01.963166 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qgkpn" event={"ID":"bf95c1a4-4c19-4f20-8014-49afdffd700b","Type":"ContainerStarted","Data":"387eaf3e6c844b4ff31e487b603fc6e58f63fe2999328816b4ff096a6d998230"} Nov 03 16:15:02.008097 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:02.007985 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qgkpn" podStartSLOduration=66.875347281 podStartE2EDuration="1m8.00796706s" totalImagesPullingTime="1.132619779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:02.005821146 +0000 UTC m=+68.224906019" watchObservedRunningTime="2026-11-03 16:15:02.00796706 +0000 UTC m=+68.227051932" Nov 03 16:15:03.974517 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:03.974473 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhgz9" event={"ID":"b125f59a-0236-430f-9c41-57402f877adc","Type":"ContainerStarted","Data":"5713489e41b0ceb1d19bdb96901668f5169f66e2d60f567ba40baa06d1c927ea"} Nov 03 16:15:03.974983 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:03.974663 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:15:04.019129 ip-10-0-129-130 kubenswrapper[2583]: I1103 16:15:04.019068 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zhgz9" podStartSLOduration=66.679799415 podStartE2EDuration="1m10.019050428s" totalImagesPullingTime="3.339251013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:04.016846711 +0000 UTC m=+70.235931582" watchObservedRunningTime="2026-11-03 16:15:04.019050428 +0000 UTC m=+70.238135300" Nov 03 16:15:04.594886 ip-10-0-129-130 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 16:15:04.612844 ip-10-0-129-130 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 16:15:04.613016 ip-10-0-129-130 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 16:15:04.613066 ip-10-0-129-130 systemd[1]: kubelet.service: Consumed 4.820s CPU time. Nov 03 16:15:04.632031 ip-10-0-129-130 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:15:04.698597 ip-10-0-129-130 kubenswrapper[6001]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:15:04.698597 ip-10-0-129-130 kubenswrapper[6001]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:15:04.698597 ip-10-0-129-130 kubenswrapper[6001]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:15:04.698597 ip-10-0-129-130 kubenswrapper[6001]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:15:04.700744 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700645 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.700744 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700742 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.700744 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700746 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.700744 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700750 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700753 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700756 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700759 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700762 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700764 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700767 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700776 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700779 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700782 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700784 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700787 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700790 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700792 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700795 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700798 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700801 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700804 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700807 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700810 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.700877 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700813 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700815 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700818 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700821 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700824 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700828 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700831 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700833 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700836 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700839 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700842 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700844 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700848 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700853 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700857 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700860 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700863 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700865 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700868 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.701369 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700872 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700876 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700879 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700882 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700885 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700887 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700890 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700893 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700896 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700898 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700901 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700904 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700906 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700909 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700912 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700914 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700917 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700919 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700922 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.701847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700925 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700928 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700931 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700935 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700937 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700943 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700946 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700949 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700951 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700954 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700956 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700959 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700961 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700964 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700968 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700971 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700974 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700978 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700982 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700985 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.702312 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700988 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700991 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700993 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700996 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.700998 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701001 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701003 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701006 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701009 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701012 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701014 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701017 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701019 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701022 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701025 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701028 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701031 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701035 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701038 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.702806 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701040 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701043 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701047 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701050 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701053 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701055 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701924 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701932 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701937 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701941 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701944 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701947 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701950 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701953 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701955 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701958 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701961 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701963 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701966 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701969 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701971 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.703270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701974 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701977 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701980 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701982 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701985 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701987 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701990 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701992 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701995 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.701999 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702001 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702008 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702011 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702014 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702016 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702019 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702022 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702025 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702027 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702030 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.703820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702033 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702036 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702038 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702041 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702044 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702046 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702049 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702051 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702054 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702056 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702059 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702062 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702065 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702067 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702070 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702072 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702075 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702078 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702081 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702083 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.704293 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702086 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702088 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702091 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702095 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702098 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702100 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702103 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702106 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702108 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702111 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702114 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702116 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702119 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702122 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702125 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702128 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702130 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702133 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702136 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702138 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.704848 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702141 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702144 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702147 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702150 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702152 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702155 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702158 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702160 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702163 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702165 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702168 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702171 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702173 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702176 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702180 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702186 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702189 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702192 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702195 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702198 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.705339 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702201 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702204 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702207 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702210 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702212 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702215 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702218 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702220 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702223 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702226 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702231 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702803 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702810 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702813 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702818 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702821 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702824 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702827 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.705826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702830 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702832 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702835 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702838 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702841 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702844 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702846 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702849 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702852 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702855 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702857 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702860 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702863 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702865 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702868 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702870 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702873 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702875 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702878 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702881 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.706250 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702884 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702887 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702889 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702892 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702894 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702897 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702899 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702902 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702906 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702909 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702911 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702914 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702916 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702919 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702922 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702924 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702927 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702930 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702933 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702935 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.706774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702938 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702941 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702943 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702946 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702948 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702951 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702953 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702956 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702958 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702961 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702963 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702966 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702968 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702972 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702975 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702977 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702980 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702982 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702985 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.707271 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702988 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702992 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702994 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702996 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.702999 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703002 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703004 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703006 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703009 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703012 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703015 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703019 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703021 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703024 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703026 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703030 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703034 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703038 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703041 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703043 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.707768 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703046 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703048 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703051 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703053 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703056 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703058 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703061 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703064 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703066 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703069 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703071 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703076 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703079 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703083 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703087 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703090 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703092 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703096 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703099 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.708248 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703101 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703202 6001 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703210 6001 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703219 6001 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703226 6001 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703234 6001 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703238 6001 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703245 6001 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703250 6001 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703254 6001 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703257 6001 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703261 6001 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703264 6001 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703267 6001 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703270 6001 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703273 6001 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703280 6001 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703283 6001 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703287 6001 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703291 6001 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703294 6001 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703298 6001 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703301 6001 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703305 6001 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:15:04.708701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703309 6001 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703313 6001 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703317 6001 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703321 6001 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703324 6001 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703327 6001 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703330 6001 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703334 6001 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703336 6001 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703341 6001 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703345 6001 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703348 6001 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703350 6001 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703354 6001 flags.go:64] FLAG: --enable-server="true" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703357 6001 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703362 6001 flags.go:64] FLAG: --event-burst="100" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703365 6001 flags.go:64] FLAG: --event-qps="50" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703368 6001 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703371 6001 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703374 6001 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703379 6001 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703382 6001 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703386 6001 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703389 6001 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703394 6001 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:15:04.709330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703397 6001 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703400 6001 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703403 6001 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703406 6001 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703409 6001 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703413 6001 flags.go:64] FLAG: --feature-gates="" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703421 6001 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703424 6001 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703427 6001 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703432 6001 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703435 6001 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703438 6001 flags.go:64] FLAG: --help="false" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703455 6001 flags.go:64] FLAG: --hostname-override="ip-10-0-129-130.ec2.internal" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703458 6001 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703461 6001 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703464 6001 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703468 6001 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703472 6001 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703475 6001 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703477 6001 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703480 6001 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703483 6001 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703487 6001 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703490 6001 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:15:04.710357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703493 6001 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703496 6001 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703499 6001 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703502 6001 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703505 6001 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703508 6001 flags.go:64] FLAG: --lock-file="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703511 6001 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703516 6001 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703519 6001 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703525 6001 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703528 6001 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703531 6001 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703535 6001 flags.go:64] FLAG: --logging-format="text" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703538 6001 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703542 6001 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703545 6001 flags.go:64] FLAG: --manifest-url="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703548 6001 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703554 6001 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703557 6001 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703562 6001 flags.go:64] FLAG: --max-pods="110" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703565 6001 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703568 6001 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703571 6001 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703574 6001 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703577 6001 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:15:04.711053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703581 6001 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703584 6001 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703591 6001 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703594 6001 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703598 6001 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703601 6001 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703604 6001 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703607 6001 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703610 6001 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703613 6001 flags.go:64] FLAG: --port="10250" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703616 6001 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703619 6001 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0538baca60d541a1d" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703623 6001 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703626 6001 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703630 6001 flags.go:64] FLAG: --register-node="true" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703633 6001 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703637 6001 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703640 6001 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703644 6001 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703646 6001 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703651 6001 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703654 6001 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703657 6001 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703660 6001 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703665 6001 flags.go:64] FLAG: --runonce="false" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703668 6001 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:15:04.711677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703671 6001 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703675 6001 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703678 6001 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703680 6001 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703684 6001 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703687 6001 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703690 6001 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703693 6001 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703696 6001 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703699 6001 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703702 6001 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703706 6001 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703709 6001 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703712 6001 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703717 6001 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703720 6001 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703723 6001 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703728 6001 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703731 6001 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703733 6001 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703738 6001 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703741 6001 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703744 6001 flags.go:64] FLAG: --v="2" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703748 6001 flags.go:64] FLAG: --version="false" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703753 6001 flags.go:64] FLAG: --vmodule="" Nov 03 16:15:04.712291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703757 6001 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.703761 6001 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703909 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703913 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703917 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703921 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703924 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703927 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703930 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703932 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703935 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703938 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703940 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703943 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703946 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703948 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703951 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703954 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703957 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703960 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.713022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703962 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703965 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703968 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703970 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703973 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703975 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703978 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703984 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703986 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703989 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703992 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703995 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.703997 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704000 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704002 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704005 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704008 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704012 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704015 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.713544 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704017 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704021 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704026 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704029 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704032 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704034 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704037 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704041 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704044 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704047 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704050 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704053 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704055 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704058 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704060 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704063 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704066 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704068 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704071 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.714001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704074 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704078 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704081 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704083 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704086 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704088 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704091 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704093 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704096 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704099 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704101 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704105 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704108 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704111 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704114 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704117 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704120 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704122 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704125 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.714459 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704127 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704130 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704133 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704135 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704138 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704140 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704143 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704145 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704148 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704151 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704153 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704156 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704159 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704161 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704166 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704168 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704171 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704173 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704176 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.714959 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704179 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704181 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704184 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704187 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704189 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704193 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704196 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704199 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704203 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704205 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704208 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704211 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.704236 6001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704279 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704283 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704288 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.715511 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704292 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704295 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704298 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704301 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704304 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704308 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704311 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704314 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704317 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704320 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704323 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704325 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704329 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704332 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704334 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704337 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704340 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704342 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704345 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704348 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.716077 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704350 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704354 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704356 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704359 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704362 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704365 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704368 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704371 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704373 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704376 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704379 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704381 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704384 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704386 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704389 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704391 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704394 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704397 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704399 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.717000 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704402 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704404 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704407 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704409 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704412 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704416 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704419 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704422 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704424 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704427 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704430 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704432 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704435 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704438 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704452 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704455 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704458 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704461 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704464 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704467 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.717822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704469 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704472 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704475 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704477 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704481 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704484 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704486 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704489 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704492 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704495 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704497 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704500 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704502 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704505 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704507 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704510 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704512 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704517 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704519 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704522 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.718455 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704524 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704527 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704530 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704533 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704535 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704538 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704540 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704543 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704546 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704548 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704551 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704554 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704557 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704560 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704562 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704566 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704568 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704571 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704573 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.719066 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704576 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704579 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704581 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704584 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.704586 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.708466 6001 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.708483 6001 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.708504 6001 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.708509 6001 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708571 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708576 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708579 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708582 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708585 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708589 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708592 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708595 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708598 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708600 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.719594 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708603 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708606 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708609 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708611 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708614 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708618 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708622 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708625 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708627 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708630 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708633 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708636 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708638 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708641 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708644 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708647 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708650 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708652 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708655 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708658 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.720040 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708660 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708663 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708666 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708669 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708671 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708674 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708677 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708679 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708682 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708684 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708687 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708690 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708692 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708695 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708697 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708700 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708703 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708706 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708710 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708712 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.720529 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708715 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708718 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708720 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708723 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708725 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708728 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708731 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708734 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708736 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708739 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708741 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708744 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708747 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708749 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708752 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708754 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708757 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708761 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708765 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.721010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708768 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708772 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708775 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708777 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708780 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708783 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708785 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708788 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708791 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708793 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708796 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708799 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708803 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708806 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708808 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708811 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708813 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708816 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708818 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708821 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.721482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708823 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708826 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708829 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708833 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708836 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708839 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708842 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708845 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708848 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708851 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708853 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708856 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708859 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708862 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708864 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708867 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.721961 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.708869 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.708899 6001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709043 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709048 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709051 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709055 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709057 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709060 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709063 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709066 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709069 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709071 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709074 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709077 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709080 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709082 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.722429 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709086 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709088 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709091 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709093 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709096 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709099 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709101 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709104 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709106 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709109 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709113 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709117 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709119 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709122 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709125 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709128 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709130 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709133 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709136 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709139 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.722865 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709141 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709144 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709147 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709149 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709152 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709155 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709208 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709777 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709782 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709785 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709788 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709791 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709794 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709797 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709799 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709803 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709805 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709808 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709811 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.723340 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709814 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709822 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709827 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709831 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709835 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709840 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709844 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709849 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709854 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709858 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709863 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709870 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709876 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709881 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709890 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709899 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709903 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709909 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709915 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709919 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.723814 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709924 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709928 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709934 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709938 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709942 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709947 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709956 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709960 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709965 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709971 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709977 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709982 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709988 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709993 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.709998 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710002 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710007 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710012 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710017 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.724327 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710026 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710030 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710034 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710038 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710043 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710048 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710052 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710057 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710064 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710068 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710073 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710077 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710086 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710091 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.710150 6001 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710266 6001 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:15:04.725029 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710274 6001 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710279 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710283 6001 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710287 6001 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710292 6001 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710301 6001 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710305 6001 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710309 6001 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710314 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710318 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710325 6001 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710332 6001 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710338 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710343 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710348 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710353 6001 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710363 6001 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710370 6001 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710375 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710380 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:15:04.725435 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710385 6001 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710389 6001 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710393 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710398 6001 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710403 6001 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710408 6001 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710412 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710416 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710420 6001 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710429 6001 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710433 6001 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710438 6001 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710458 6001 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710463 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710467 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710471 6001 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710476 6001 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710480 6001 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710485 6001 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:15:04.726011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710489 6001 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710494 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710498 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710506 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710511 6001 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710515 6001 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710519 6001 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710524 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710535 6001 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710540 6001 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710546 6001 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710550 6001 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710555 6001 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710559 6001 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710563 6001 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710567 6001 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710577 6001 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710601 6001 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710606 6001 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710609 6001 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:15:04.726512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710611 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710614 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710617 6001 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710620 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710623 6001 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710627 6001 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710630 6001 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710634 6001 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710639 6001 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710649 6001 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710790 6001 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710802 6001 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710808 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710813 6001 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710818 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710822 6001 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710827 6001 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710831 6001 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710838 6001 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:15:04.727015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710842 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710847 6001 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710851 6001 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710858 6001 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710862 6001 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710866 6001 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710871 6001 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710875 6001 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710879 6001 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710883 6001 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710889 6001 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710893 6001 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710897 6001 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710902 6001 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710906 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710911 6001 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710915 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710920 6001 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710924 6001 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:15:04.727611 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710928 6001 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710932 6001 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710937 6001 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710941 6001 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710945 6001 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710950 6001 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710954 6001 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:04.710961 6001 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.711094 6001 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712156 6001 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712229 6001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712428 6001 server.go:1014] "Starting client certificate rotation" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712578 6001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712630 6001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:54 +0000 UTC" deadline="2028-05-01 13:48:02.558903053 +0000 UTC" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712669 6001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13077h32m57.846235576s" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.712936 6001 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:15:04.728082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.714079 6001 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:15:04.728493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.715317 6001 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:15:04.728493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.717925 6001 log.go:25] "Validated CRI v1 image API" Nov 03 16:15:04.728493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.718674 6001 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:15:04.728493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.721199 6001 fs.go:135] Filesystem UUIDs: map[33c7b21c-c774-46ea-a28e-24b2a9c46627:/dev/nvme0n1p3 3c4d8ae7-1fe2-41d2-aadf-90f59d2fe2d9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 03 16:15:04.728667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.721223 6001 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c/userdata/shm major:0 minor:519 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e/userdata/shm major:0 minor:502 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6/userdata/shm major:0 minor:464 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86/userdata/shm major:0 minor:515 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d516a932e9ab0922cdcd8448881e85e124db634da2d74adec15777400d30e7ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d516a932e9ab0922cdcd8448881e85e124db634da2d74adec15777400d30e7ab/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05/userdata/shm major:0 minor:516 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0811b831-8592-4677-aa31-cd58433a3f5c/volumes/kubernetes.io~projected/kube-api-access-phvqh:{mountpoint:/var/lib/kubelet/pods/0811b831-8592-4677-aa31-cd58433a3f5c/volumes/kubernetes.io~projected/kube-api-access-phvqh major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0de08296-dbf4-45f3-8dc4-33cbf8de0658/volumes/kubernetes.io~projected/kube-api-access-tphwm:{mountpoint:/var/lib/kubelet/pods/0de08296-dbf4-45f3-8dc4-33cbf8de0658/volumes/kubernetes.io~projected/kube-api-access-tphwm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~projected/kube-api-access-h2sdp:{mountpoint:/var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~projected/kube-api-access-h2sdp major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/config-reloader/2 major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/prometheus/15 major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~empty-dir/config-out major:0 minor:462 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/kube-api-access-f798l:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/kube-api-access-f798l major:0 minor:463 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/tls-assets major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/config major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:458 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:459 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/web-config major:0 minor:457 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~projected/kube-api-access-569mw:{mountpoint:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~projected/kube-api-access-569mw major:0 minor:501 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~projected/kube-api-access-87cd7:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~projected/kube-api-access-87cd7 major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~projected/kube-api-access-tzt6k:{mountpoint:/var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~projected/kube-api-access-tzt6k major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~projected/kube-api-access-gqmvz:{mountpoint:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~projected/kube-api-access-gqmvz major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~projected/kube-api-access-9rr6v:{mountpoint:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~projected/kube-api-access-9rr6v major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~projected/kube-api-access-7nql7:{mountpoint:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~projected/kube-api-access-7nql7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/alertmanager/9 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/config-reloader/6 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~empty-dir/config-out major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/kube-api-access-s8lz6:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/kube-api-access-s8lz6 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/tls-assets major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/config-volume major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/web-config major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~projected/kube-api-access-pwvp5:{mountpoint:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~projected/kube-api-access-pwvp5 major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a163d0db-7c08-45c9-bf83-6132d280a72a/volumes/kubernetes.io~projected/kube-api-access-vbb5t:{mountpoint:/var/lib/kubelet/pods/a163d0db-7c08-45c9-bf83-6132d280a72a/volumes/kubernetes.io~projected/kube-api-access-vbb5t major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:513 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8555d4b-9866-49fa-b635-d9b8e26f8b3b/volumes/kubernetes.io~projected/kube-api-access-tjcx5:{mountpoint:/var/lib/kubelet/pods/a8555d4b-9866-49fa-b635-d9b8e26f8b3b/volumes/kubernetes.io~projected/kube-api-access-tjcx5 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b125f59a-0236-430f-9c41-57402f877adc/volumes/kubernetes.io~projected/kube-api-access-bj5mm:{mountpoint:/var/lib/kubelet/pods/b125f59a-0236-430f-9c41-57402f877adc/volumes/kubernetes.io~projected/kube-api-access-bj5mm major:0 minor:514 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~projected/kube-api-access-rp7gg:{mountpoint:/var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~projected/kube-api-access-rp7gg major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~projected/kube-api-access-xxqrw:{mountpoint:/var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~projected/kube-api-access-xxqrw major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:508 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~projected/kube-api-access-jjntj:{mountpoint:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~projected/kube-api-access-jjntj major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~projected/kube-api-access-sbhxv:{mountpoint:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~projected/kube-api-access-sbhxv major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/debced67-f8f8-480e-a1bc-4add3059fd0e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/debced67-f8f8-480e-a1bc-4add3059fd0e/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~projected/kube-api-access-c9rzx:{mountpoint:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~projected/kube-api-access-c9rzx major:0 minor:267 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~projected/kube-api-access-p6928:{mountpoint:/var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~projected/kube-api-access-p6928 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~projected/kube-api-access-mvqwz:{mountpoint:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~projected/kube-api-access-mvqwz major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6cd40ea-b486-4d7d-a8ec-49d153976a09/volumes/kubernetes.io~projected/kube-api-access-fj497:{mountpoint:/var/lib/kubelet/pods/f6cd40ea-b486-4d7d-a8ec-49d153976a09/volumes/kubernetes.io~projected/kube-api-access-fj497 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fde7e318-0d30-4892-81f1-3e4f19c782f5/volumes/kubernetes.io~projected/kube-api-access-ng4vr:{mountpoint:/var/lib/kubelet/pods/fde7e318-0d30-4892-81f1-3e4f19c782f5/volumes/kubernetes.io~projected/kube-api-access-ng4vr major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/52f9af52e373f009f273105c1baf4e7dfb24ec171bbbc9f21109f867a98370d6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/55bc1e54fe26450d98d5619a846bf9c456c572e295ffd8c41ecf84233c7ea5ac/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/67eca2b87428e51fabb55c282c40147104b25b5ffc9c5e2afae16bb72892ebf1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/eea230e3a6ca179276043d3f5fb9275b749f365b286df850dfb902ef408806ac/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/88d174716bc869f3a2c1e65c875a0a98d5f145a5e52a492bacf15db87e7d7465/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/00352284178cd7efdf62214971a7a82fed36e97d44db042193ef40fea3ebaaf8/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/c5e7b7346823b796384a70fbe6b4b7fc3b2936d5870752a2156d0d3337010597/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4b3c12b29deb083d669714435df46621311819479e26411dc81e82888fb50e52/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/523338c795a9510de5bec813087f7fbde2256ff6daed430cbca27d5c67682465/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/47c70f05d97250685378eac3a593e0c237beba29661232f96b8bcdd44d4ed2c7/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/bb7c20f30752c33d42a6a11631c2f5e87a086ad9bb25a4f6536d142cbffee424/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/5181c116658178c1ca03f41afed800bda21895d8a3f75aeea57efcaf6112e7bf/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/00f65224aca8f15bc13c0d81b298cb71a0f72226732ec1b0f3fbeb0bc0fed675/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/32810817b2190199b954d1ac473a2139946cb20b63290b5a91b5587d608f8188/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f567e4f9a7a1ad76971be33b99eb11c27693d69e679c7ff909a9c0c92264a8c6/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c14dee243a7e7fb38af5b3058aa132bc9fc6df4806cb94b773cf11b7b20cc6f3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/29094a43fc63e22e3c18e9ab86fb88b1e8f1aed5422c7b3c538c1e2add14b59e/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d3354d8e49fd10cea279cd8bb0b1281d0fc01e1a145ed5948b76dbc486e3e6f2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/842ac64004b2587a89c1ba68a1208ebf67bc21ee4a0b4d1ccbfd7a558940be1a/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/44bf25640a9f26e60dc0ee3129344a2ddcd2ccbf6aabb6bfdee0959bf55ca8b9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5f81532ddd47f80680875c55e2b4a08a3aba639258d0fb47115bfef087f32da8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/1d529219d8924dbf191952c1890206d7383d67e97148b7d020a731b19dc7dda1/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/7b67cb6bf7209d73742a26bc0f529865e22348ede9ee1bf107988e4ba1a5a4f8/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/133e005346ec2e637aa9ea0241e8e9d4db64ec764cea2e47f8f50731b7f9da73/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/d99d5708a5a51d02435e4e5c8a9e86ebb67880bac049870f431ac6b423b2bfa5/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/f4fd60cce5354f067c6604fc0f12caddab760b7563a794f0742e1835e80389bf/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/a3967512986711088bcaea562c1db601b9fd67c2d892930bae8582033cad1f89/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/e786f05229ddd9c12f35b143376277ab9e00e0567a5d3c5570749f65b13fe471/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/7478c717c73aaccc10a2c98e301040dee73681ff2cc9ab082dd58e67e731b711/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c57fcf3d6d46721c3725f26ba38a8ecb6512a24aaf87fa49f9d164a39e3ac75e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/bcc065588586875d386090015acd994717eff4ecebe156c27973d5a2f37c44e7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/e9ecb2ea3af3f2ad2f8c902fa45a7385e56ee76ef6a4848be40bb8609f7ea2c0/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/97751cd2c150d69f9c48d29efb845f5529d92012413969b55b0a9340b9e9de64/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/194d8fa3a3562ce17df7a4136d0194a7ef7ca11226b9885a2f1f1710bab77907/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/c0804f4edb663ae211a24b4c04216a943515c2809d32d41eca3bab4d8980a5b0/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/c47a412906e03668ca8eaeaf945b5047a1e0dae5cf17d24df19f0d4313c2a226/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/61b55a316225453fa2f7280b0c95093c85839c6c10be53798fb41e7ee98d70a3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/9fb81dc7bc7687da654965e9af7eb5be5552eff6c476c8b7ae49e6bbfc8af4c7/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/dae640ecd9dfd925da85496e5bf83576c8d032c73a9346d536600c2a0b8d1b64/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/c6f8f8bc6a68b4fe6367bbe0cc48dfe7079657f7cd27bfbad82f47db6a0d3d15/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/b20ffa7cf5a2d55f4bcfa2de4e14b65672d160f5d1374fb8403a01fb1e4c741f/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/3e76479597aa37c4d796822adbadb01557556857a1a56d8055a60c08097068c8/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/ffa0f408ef168caffa4403ad3eaa429ce7a5d6d9bfedd8b2305e2e5ed60fac07/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/21ab339aab8cfe0d1930e9ffe293842e0721a9e0332acef46fc3f3fde77e2ce8/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/ef98d9ab14a473b6404f9b1c06d1983f708af4f456e64dea0a4097917f5e004d/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/bd3f4983e247c05fc55e684fdefd11777efa3b77a5d9757efc9b0a0dcd97cfb6/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/5d5d41920a035c31795389c9c2d7904b4607ee9bd7f7e2e1a97a2e6443daeb81/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/c708d60168cad42326a9b1f123d10c4da7bbd431716a0cde0894ae4706a3d641/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/7b1caca8692e2fe37430781248f0a93e0ce614897637e667d7e4caa8f423a174/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/bef9854a43fc6b75acafcfecd62744b872fe6d19b27a11542e560376b710644d/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/68f96930588101a626a4769c10f00c3c859077794c1fb7f87ee4a0f84593ad78/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/a5809a06222c1d5e76dd5a5a390708e7d95dd3384cad0ec54a7f87ffd1e88cf4/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/ab8ea25ac7416d06181c844d697ff080057666880c43b448e1bae7aee75bb869/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/1613f01b090fdc1fcedec91dfd83da666269382a77b36a51469cc0fd30ef0e30/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/a638bd81237cacecaf481fc7c78428f0201a4a5f6b9766dd4654dd86884c8ac6/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/1a54dc2d9284858aad881530748ad00ffa2f9ac718fbf6c980a15035b6318bc4/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/57a801e13f021c75eaf4caff9ca187b5a8a6efa7ac531bae34cce7ac91a83468/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/7e3a44cd372e778f4cd9645df0b2ae1294161f99300daecf90a15df376b13122/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/7ee636f04fde828618959490d8cb2ced464899dce7658c44fe4f5d64c832d0d5/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/99171f82c81954d8afcb841acbc27f625feecd6a7c9904bad4f47759561a955f/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/29f7e8dfc28049a6bac2dec5b85fd3ee64b73881c2ff5a5ae9390dffae2b7455/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/56905edb5bb791d3c2a736e5e278b2f612121387f370690a6acb645696b83f51/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/28a18c0c3a88d9af2de1c0d6c27b3286ac4497a0e92d88542697dc597b4b47bb/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/e71b9b6146ef934167ccadeec5a5a2bc5809c1e572e2f837166944b6fb9d4f72/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/de11a17d28e10b4a60c7d1f03f2355c5f020cff0d1b6b83060b1ac973b518fa5/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/ea8c3c4370da3d3e2e2698e5ec56db923fa34373063f718877adcfee0a4788d0/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/a602e4e165e39f9a02345a312e88ea64481c3a1428e7ff9cb085c3d6413722d2/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/34837eb4a7357f038c4bdf4ca6b75c310642d25476b833a380f4d7b5879c5029/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/56fb4df33d33e298301933116e92160ffcf45f8250baca6a25c39b0a6394cc53/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/3ed34914039668e6a03815189fc4773a3ea2c13ad9406624914437924c7c0072/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/7df88ad5999be563233aad0cb9eeb89c532c801f8d4961d4216888d86a150b1b/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/99d410121c03cb2e718e87b0488fbbd6f9e120f8361ba96b791ec43f2f2e52d2/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/775d2672d77a0059938c6bf2e60c8dc5e35f3c88a1cde853edc618e66da567c9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-521:{mountpoint:/var/lib/containers/storage/overlay/923723d0506b6d7f4942fe28280b8bc17749f9e0af3b172b1b469f21be362ea5/merged major:0 minor:521 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/bde3333590c4622105815183f65c07b1b7255c4ea1aad4dc6506f9e52b449cc6/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-525:{mountpoint:/var/lib/containers/storage/overlay/ee30c416accf88db66ae5ab691ae16d7b3770d126ca70c7a5a79a702847cc307/merged major:0 minor:525 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/394a8edb70c117f177e642de6b1e73381425f0c8d70acec19c2712c7442134b1/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-529:{mountpoint:/var/lib/containers/storage/overlay/a7c4be5c20e0ea1aaaaea874525c71d24f2e7aa4f9ac6833b8dca7925a3f4704/merged major:0 minor:529 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fb5aa532485007fec36fb3614ff1c5fd84a386c9e1bf90933381d31f372db7be/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-535:{mountpoint:/var/lib/containers/storage/overlay/12f4cc69133444ee4e83b8301d0a504581f9685537d9fe96e0c3cabf3339772a/merged major:0 minor:535 fsType:overlay blockSize:0} overlay_0-540:{mountpoint:/var/lib/containers/storage/overlay/096c43abd86f87eaa22980dfd2af22415b05c585e7751b0233e364939edb4a04/merged major:0 minor:540 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/66c70fef2aa58381cb65612b4d53c9b9f3b9f5f5264f7e24c7b3cc3df74fbd03/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/1805386ad0c26f0a9b372be0ac9e850d720028e02065abf2116d6daf6771a09f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/eb6f8a14bd709426a3290b0f06be7af50242a849e61c42b0a07c9b3ae1080dd2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ab777fb0083bada21f1cd2c43ab9385d6adc304d4f17bf9e9fed28ed2e382748/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/3df34cbad86411171e7f90fd6bb840af24f178a8423c365b0d46cafccb363ef9/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-91:{mountpoint:/var/lib/containers/storage/overlay/ebded2e23871fd71b98a474f9cf3343865c884c1880af79da5ff2b3f1d06fc7f/merged major:0 minor:91 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9578cacaa959380137e5dc034a20ba3f0ca49f860e8748c810fc3e09510bafcb/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/452c2fd97df305aa51b3bddd3faf8519ffc7721bbf3093a2df5fd47e769a2022/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 16:15:04.731250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.730734 6001 manager.go:217] Machine: {Timestamp:2026-11-03 16:15:04.729802705 +0000 UTC m=+0.063016275 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097547 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28984d6ec630f475e4414e41de3dcd SystemUUID:ec28984d-6ec6-30f4-75e4-414e41de3dcd BootID:c86926e4-deb7-4436-bc6d-6de524aafa25 Filesystems:[{Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:457 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b125f59a-0236-430f-9c41-57402f877adc/volumes/kubernetes.io~projected/kube-api-access-bj5mm DeviceMajor:0 DeviceMinor:514 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b57f6e80dd4112988c600d05372a48ffa4de0e0e32c0be524aab52af1d940652/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25a052b19270a0783564bfeb36bc2d2b3c3922d0891293444fb70df9cd312c2c/userdata/shm DeviceMajor:0 DeviceMinor:519 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~projected/kube-api-access-h2sdp DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3fe2a0360f50a7d7280a17d5fdb96d7f0b48bec59aa2bebf01867d5d5c61d05/userdata/shm DeviceMajor:0 DeviceMinor:516 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d71a69b7b27f6433e398ce0780cdc3c3ea57fded37a1f5327568ef1540e1f3f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:265 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a163d0db-7c08-45c9-bf83-6132d280a72a/volumes/kubernetes.io~projected/kube-api-access-vbb5t DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:459 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-540 DeviceMajor:0 DeviceMinor:540 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6cd40ea-b486-4d7d-a8ec-49d153976a09/volumes/kubernetes.io~projected/kube-api-access-fj497 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~projected/kube-api-access-7nql7 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f6f83f1faa8f7321879cde114bb479662a0ce26fc02aecb7efef58fcf701378/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/458d4bd3b41560242f81f6e18640e8f8d247ed1758d0c31b832af9a2244ab94e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2472ce5c59e4e098d3418777b891f77afe8dcc97c71e1d0c555dcee1146cf04/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:457 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fde7e318-0d30-4892-81f1-3e4f19c782f5/volumes/kubernetes.io~projected/kube-api-access-ng4vr DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~projected/kube-api-access-s8lz6 DeviceMajor:0 DeviceMinor:304 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8555d4b-9866-49fa-b635-d9b8e26f8b3b/volumes/kubernetes.io~projected/kube-api-access-tjcx5 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/591381c1d6b760f3f6d44f504e00c8e1bbde7fcee73441e520403020f0879ba6/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31b00ba6709a79994a86c49f37455ba4415d25afc429654d5de94f0de0fdb48e/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:455 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-91 DeviceMajor:0 DeviceMinor:91 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~projected/kube-api-access-9rr6v DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5de1954582c6dbe7020df8163f3dffd1c78c403d5f09a4a8cba829a1ff0ee40/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~projected/kube-api-access-gqmvz DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:462 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~projected/kube-api-access-569mw DeviceMajor:0 DeviceMinor:501 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e0f1a2420df832a378dc5b31ebc1ce2c93e240e769e8ee0c214201784a356f5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f18098b-6d38-4cba-85ed-5e110a988df8/volumes/kubernetes.io~projected/kube-api-access-tzt6k DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:445 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:513 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-521 DeviceMajor:0 DeviceMinor:521 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~projected/kube-api-access-xxqrw DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-535 DeviceMajor:0 DeviceMinor:535 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16969736b53928b07d6b15551f96f76b71a4881382c877ccb70ce51d89c2711a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b14267fde1fa10cb98355d810867bdfa208abdf76c6c51bfc9146c0ccde054f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f92266d0693c2ea189132eb02faecff0c6073cf68470d8bd074ea441d99cb83b/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5c3fd164ae74a9e1a63b2c455afac5d56696eae5ab3ff36e6bf0670360eab89/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc68039f4b04306babf5f24e8ad82511e19852e3f78b1fb0e98db94f08dc9a86/userdata/shm DeviceMajor:0 DeviceMinor:515 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/746b87e4-de05-417f-82ba-53e481c27611/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d4f2426234b1145fd74b244c5efdd5a5320bf62d5dba59bc814f660724a7af1/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:378 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/kube-api-access-f798l DeviceMajor:0 DeviceMinor:463 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/debced67-f8f8-480e-a1bc-4add3059fd0e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~projected/kube-api-access-c9rzx DeviceMajor:0 DeviceMinor:267 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bc21144a-4ab2-4b38-b190-54a63a17c90c/volumes/kubernetes.io~projected/kube-api-access-rp7gg DeviceMajor:0 DeviceMinor:379 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:496 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bf95c1a4-4c19-4f20-8014-49afdffd700b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:508 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:454 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:460 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d16e7255-2933-413f-9114-4b1e2f93d744/volumes/kubernetes.io~projected/kube-api-access-sbhxv DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/caa2b183-e4e8-41fc-85e4-52293e0add64/volumes/kubernetes.io~projected/kube-api-access-jjntj DeviceMajor:0 DeviceMinor:262 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d516a932e9ab0922cdcd8448881e85e124db634da2d74adec15777400d30e7ab/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/deeed625-5069-4da6-8482-2c8ca7c4ff88/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:266 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-529 DeviceMajor:0 DeviceMinor:529 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0811b831-8592-4677-aa31-cd58433a3f5c/volumes/kubernetes.io~projected/kube-api-access-phvqh DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:457 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62976dc8b697e37463c06a2fcbe99a59debacf8af96de728008b51668cf166a6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:260 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~projected/kube-api-access-mvqwz DeviceMajor:0 DeviceMinor:375 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:500 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1995da86606e121e5b3b94f38aaec90c2b2ebb0934be89dc4403dbddb0328d56/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0de08296-dbf4-45f3-8dc4-33cbf8de0658/volumes/kubernetes.io~projected/kube-api-access-tphwm DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:444 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:456 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e757f958-921a-4cc9-9735-c3f5594b55d9/volumes/kubernetes.io~projected/kube-api-access-p6928 DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:461 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6/userdata/shm DeviceMajor:0 DeviceMinor:464 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-525 DeviceMajor:0 DeviceMinor:525 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/volumes/kubernetes.io~projected/kube-api-access-87cd7 DeviceMajor:0 DeviceMinor:344 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:373 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:458 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:457 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e/userdata/shm DeviceMajor:0 DeviceMinor:502 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26796971-9faf-4381-8f8b-23db59a37fcc/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a13a995e-bff6-4420-b8d3-4d2f107c2fa4/volumes/kubernetes.io~projected/kube-api-access-pwvp5 DeviceMajor:0 DeviceMinor:261 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3e957f4a0fb52daba271fb0e53d3e84259166b0e7f80e26df676ac7c589b790/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1da53ff5ca617cd MacAddress:02:32:be:81:9e:36 Speed:10000 Mtu:8901} {Name:25a052b19270a07 MacAddress:12:75:ae:6b:60:40 Speed:10000 Mtu:8901} {Name:31b00ba6709a799 MacAddress:ce:16:00:2e:ad:ca Speed:10000 Mtu:8901} {Name:437c0c5e594cec7 MacAddress:5e:5c:3a:a2:ee:77 Speed:10000 Mtu:8901} {Name:591381c1d6b760f MacAddress:a6:15:22:c6:4a:cc Speed:10000 Mtu:8901} {Name:685c83415098532 MacAddress:ae:a8:c8:50:f8:ee Speed:10000 Mtu:8901} {Name:8f62bec151afa46 MacAddress:c2:c4:36:30:2d:2e Speed:10000 Mtu:8901} {Name:9e39ef99d585e45 MacAddress:06:58:22:e0:b2:40 Speed:10000 Mtu:8901} {Name:a3e957f4a0fb52d MacAddress:4e:7b:55:bf:13:ab Speed:10000 Mtu:8901} {Name:b57f6e80dd41129 MacAddress:de:33:57:bf:13:82 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c5:cb:c2:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:aa:6b:85:17:92:ac Speed:0 Mtu:8901} {Name:c5c3fd164ae74a9 MacAddress:9a:9a:e4:e6:d6:4a Speed:10000 Mtu:8901} {Name:cc68039f4b04306 MacAddress:b2:c2:aa:62:76:00 Speed:10000 Mtu:8901} {Name:d516a932e9ab092 MacAddress:2a:fa:df:bf:4d:98 Speed:10000 Mtu:8901} {Name:e3fe2a0360f50a7 MacAddress:0a:db:d6:75:33:0f Speed:10000 Mtu:8901} {Name:e5de1954582c6db MacAddress:f6:b3:d3:2e:0a:2a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c5:cb:c2:69 Speed:0 Mtu:9001} {Name:f92266d0693c2ea MacAddress:5a:84:5f:d9:ab:6c Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1a:a5:56:32:d3:e4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9e:17:e7:32:e8:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:15:04.731250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731255 6001 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:15:04.731627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731309 6001 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:15:04.731627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731604 6001 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:15:04.731768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731630 6001 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-130.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:15:04.731813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731770 6001 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:15:04.731813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731778 6001 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:15:04.731813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731789 6001 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:15:04.731813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731803 6001 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:15:04.731920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.731912 6001 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:15:04.732079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732069 6001 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:15:04.732183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732175 6001 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:15:04.732220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732212 6001 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:15:04.732249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732223 6001 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:15:04.732249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732233 6001 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:15:04.732249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732243 6001 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:15:04.732333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.732251 6001 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:15:04.733218 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.733204 6001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:15:04.733218 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.733217 6001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:15:04.733786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.733760 6001 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:15:04.736725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.736708 6001 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:15:04.736725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.736722 6001 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:15:04.736917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.736826 6001 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 16:15:04.737069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737057 6001 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:15:04.737106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737087 6001 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:15:04.737189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737180 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:15:04.737220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737194 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:15:04.737220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737201 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:15:04.737220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737206 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:15:04.737220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737212 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:15:04.737220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737218 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737230 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737236 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737243 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737249 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737257 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737266 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737292 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:15:04.737349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.737298 6001 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:15:04.740828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.740812 6001 server.go:1263] "Started kubelet" Nov 03 16:15:04.740918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.740870 6001 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:15:04.740984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.740919 6001 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:15:04.741041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.740998 6001 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:15:04.741419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.741397 6001 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:15:04.741428 ip-10-0-129-130 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:15:04.742151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.742131 6001 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:15:04.746650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.746583 6001 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:15:04.747243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.747222 6001 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:15:04.747243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.747244 6001 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:15:04.747426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.747304 6001 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:15:04.748153 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:04.748125 6001 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-130.ec2.internal\" not found" Nov 03 16:15:04.748253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.748162 6001 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:15:04.748385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.748341 6001 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:55 +0000 UTC" deadline="2028-04-05 19:50:19.082845145 +0000 UTC" Nov 03 16:15:04.748436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.748389 6001 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12459h35m14.334459437s" Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.749852 6001 factory.go:55] Registering systemd factory Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.749877 6001 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750120 6001 factory.go:153] Registering CRI-O factory Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750132 6001 factory.go:223] Registration of the crio container factory successfully Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750189 6001 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750204 6001 factory.go:103] Registering Raw factory Nov 03 16:15:04.750277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750215 6001 manager.go:1199] Started watching for new ooms in manager Nov 03 16:15:04.750698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.750564 6001 manager.go:319] Starting recovery of all containers Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751054 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751096 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751113 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f18098b-6d38-4cba-85ed-5e110a988df8" volumeName="kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751128 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751140 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746b87e4-de05-417f-82ba-53e481c27611" volumeName="kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751153 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0811b831-8592-4677-aa31-cd58433a3f5c" volumeName="kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751166 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751178 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc21144a-4ab2-4b38-b190-54a63a17c90c" volumeName="kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751195 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751211 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751223 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751236 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde7e318-0d30-4892-81f1-3e4f19c782f5" volumeName="kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751248 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde7e318-0d30-4892-81f1-3e4f19c782f5" volumeName="kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751262 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751278 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751307 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de08296-dbf4-45f3-8dc4-33cbf8de0658" volumeName="kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751321 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751335 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751348 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751360 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751373 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751424 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16e7255-2933-413f-9114-4b1e2f93d744" volumeName="kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751438 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751470 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751482 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26796971-9faf-4381-8f8b-23db59a37fcc" volumeName="kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751496 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26796971-9faf-4381-8f8b-23db59a37fcc" volumeName="kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751509 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751523 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a163d0db-7c08-45c9-bf83-6132d280a72a" volumeName="kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751543 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751555 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26796971-9faf-4381-8f8b-23db59a37fcc" volumeName="kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751576 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751592 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f18098b-6d38-4cba-85ed-5e110a988df8" volumeName="kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751605 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751621 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751634 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de08296-dbf4-45f3-8dc4-33cbf8de0658" volumeName="kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751648 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751660 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751672 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751685 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e757f958-921a-4cc9-9735-c3f5594b55d9" volumeName="kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751698 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0811b831-8592-4677-aa31-cd58433a3f5c" volumeName="kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751711 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746b87e4-de05-417f-82ba-53e481c27611" volumeName="kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751725 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751738 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751752 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751765 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de08296-dbf4-45f3-8dc4-33cbf8de0658" volumeName="kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy" seLinuxMountContext="" Nov 03 16:15:04.751746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751779 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751792 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751804 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751817 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751836 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a27eb936-8e3e-4bfb-8c6a-05c184ee89ea" volumeName="kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751848 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16e7255-2933-413f-9114-4b1e2f93d744" volumeName="kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751860 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6cd40ea-b486-4d7d-a8ec-49d153976a09" volumeName="kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751872 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751884 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751897 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16e7255-2933-413f-9114-4b1e2f93d744" volumeName="kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751910 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="debced67-f8f8-480e-a1bc-4add3059fd0e" volumeName="kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751934 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6cd40ea-b486-4d7d-a8ec-49d153976a09" volumeName="kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751948 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fde7e318-0d30-4892-81f1-3e4f19c782f5" volumeName="kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751960 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751973 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751986 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.751999 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752011 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752025 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752038 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13a995e-bff6-4420-b8d3-4d2f107c2fa4" volumeName="kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752055 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e757f958-921a-4cc9-9735-c3f5594b55d9" volumeName="kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752073 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752085 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752098 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16e7255-2933-413f-9114-4b1e2f93d744" volumeName="kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752112 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752125 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752139 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752153 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13a995e-bff6-4420-b8d3-4d2f107c2fa4" volumeName="kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752166 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752179 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752193 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752207 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752220 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752233 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f18098b-6d38-4cba-85ed-5e110a988df8" volumeName="kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752247 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752260 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752275 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752288 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752302 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752316 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752328 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13a995e-bff6-4420-b8d3-4d2f107c2fa4" volumeName="kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752341 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752355 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752367 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752379 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13a995e-bff6-4420-b8d3-4d2f107c2fa4" volumeName="kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752392 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf95c1a4-4c19-4f20-8014-49afdffd700b" volumeName="kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752405 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752417 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8555d4b-9866-49fa-b635-d9b8e26f8b3b" volumeName="kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752431 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26796971-9faf-4381-8f8b-23db59a37fcc" volumeName="kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752459 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752472 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752489 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc21144a-4ab2-4b38-b190-54a63a17c90c" volumeName="kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752502 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d16e7255-2933-413f-9114-4b1e2f93d744" volumeName="kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752514 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752528 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752540 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752553 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752566 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752579 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752591 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752605 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e84b0c-2265-41de-ad75-1d9329e1dc21" volumeName="kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752617 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752630 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752643 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="746b87e4-de05-417f-82ba-53e481c27611" volumeName="kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752656 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a163d0db-7c08-45c9-bf83-6132d280a72a" volumeName="kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752669 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="efef1ee8-f44f-4b0f-9c5d-de4024cfb8da" volumeName="kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752682 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0de08296-dbf4-45f3-8dc4-33cbf8de0658" volumeName="kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752713 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54" volumeName="kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752729 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752742 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="391a45b7-3643-4fa0-994f-dc590a257672" volumeName="kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752757 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752770 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deeed625-5069-4da6-8482-2c8ca7c4ff88" volumeName="kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752784 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46511848-529c-41d9-a9ae-d09fa48ddd3d" volumeName="kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752797 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752811 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752824 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b125f59a-0236-430f-9c41-57402f877adc" volumeName="kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752836 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68a16c09-5008-44a0-b655-7b1f12ab2317" volumeName="kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752849 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752862 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92928fd5-df80-494a-b99f-1e675a190dea" volumeName="kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752875 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="debced67-f8f8-480e-a1bc-4add3059fd0e" volumeName="kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752887 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f18098b-6d38-4cba-85ed-5e110a988df8" volumeName="kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752900 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf95c1a4-4c19-4f20-8014-49afdffd700b" volumeName="kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752912 6001 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="caa2b183-e4e8-41fc-85e4-52293e0add64" volumeName="kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752928 6001 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:15:04.753287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.752936 6001 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:15:04.779381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.779339 6001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:15:04.780682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.780656 6001 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:15:04.780682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.780682 6001 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:15:04.780901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.780710 6001 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:15:04.780901 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:04.780750 6001 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 16:15:04.786910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.786884 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:15:04.787495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.787471 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:15:04.788375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.788343 6001 generic.go:356] "Generic (PLEG): container finished" podID="45286eece609f171819d9469f4da982c" containerID="79ae7886b4439f3fc1f530d84ed615e72bc323e727c000d493d9062ec9f8204c" exitCode=0 Nov 03 16:15:04.802955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.802798 6001 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:15:04.803496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.803475 6001 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 16:15:04.804064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.804045 6001 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:15:04.804786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.804763 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:15:04.805137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.805108 6001 generic.go:356] "Generic (PLEG): container finished" podID="d16e7255-2933-413f-9114-4b1e2f93d744" containerID="b454a445f22e12b81c1b6cdf4aacea05b5fa6811d9aca8f78f9e09357c41d9dd" exitCode=1 Nov 03 16:15:04.823519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823490 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="9c0d32e504812a831ea62ae73ce1e610c01c741c3c244abc785c7d7b70dc706e" exitCode=0 Nov 03 16:15:04.823519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823515 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="a3bc1e4c80e3faed59a4789538f5ef1034e41ca5da26d8a9148764ae12f7b4c3" exitCode=0 Nov 03 16:15:04.823519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823522 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="ed44bcd3b27f12506a3f9d1591ad76333d0e979b872f65314bf70b74e71788bf" exitCode=0 Nov 03 16:15:04.823749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823528 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="2ee92c9301d17fbb20ae09ff9f99683d7f5902e0c91583b0e8f30b1f9a918c6b" exitCode=0 Nov 03 16:15:04.823749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823534 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="efb691bd19bd4ab7c4bc80f295b258d9ed5eda0a29ec700cca8bfb24f4666f01" exitCode=0 Nov 03 16:15:04.823749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.823539 6001 generic.go:356] "Generic (PLEG): container finished" podID="0de08296-dbf4-45f3-8dc4-33cbf8de0658" containerID="25518ec31b9a790137628fd39b6769632dc3febc2210704ed88a9575574fdc0a" exitCode=0 Nov 03 16:15:04.828353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.828326 6001 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:15:04.829099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.829078 6001 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:15:04.835882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.835849 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" exitCode=0 Nov 03 16:15:04.838022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.838000 6001 generic.go:356] "Generic (PLEG): container finished" podID="caa2b183-e4e8-41fc-85e4-52293e0add64" containerID="b3adc34df20f0a7839a2e9ac3c0d631ee7da38b40b54f741357176b0e9f254e5" exitCode=0 Nov 03 16:15:04.851404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.851335 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496" exitCode=0 Nov 03 16:15:04.878686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.878667 6001 manager.go:324] Recovery completed Nov 03 16:15:04.880823 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:04.880800 6001 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 16:15:04.908939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.908902 6001 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:15:04.908939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.908923 6001 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:15:04.908939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.908950 6001 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909124 6001 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909140 6001 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909167 6001 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909176 6001 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909184 6001 policy_none.go:50] "Start" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909198 6001 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:15:04.909280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909212 6001 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:15:04.909626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909395 6001 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 16:15:04.909626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909407 6001 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 16:15:04.909626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.909415 6001 policy_none.go:44] "Start" Nov 03 16:15:04.914675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.914639 6001 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:15:04.917570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.917550 6001 manager.go:341] "Starting Device Plugin manager" Nov 03 16:15:04.918652 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:04.917581 6001 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:15:04.918652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.918650 6001 server.go:87] "Starting device plugin registration server" Nov 03 16:15:04.918962 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.918948 6001 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:15:04.919019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.918961 6001 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:15:04.919076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.919054 6001 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:15:04.919208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.919170 6001 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:15:04.919335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.919219 6001 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:15:04.919335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.919225 6001 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:15:04.919335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.919270 6001 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:15:04.919203809Z","UUID":"f4917172-f277-490b-b05f-b5b3e0c64059","Handler":null,"Name":"","Endpoint":""} Nov 03 16:15:04.920861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.920841 6001 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:15:04.920952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:04.920872 6001 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:15:04.921050 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:04.921022 6001 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:15:05.019675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.019632 6001 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:15:05.023050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.023020 6001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:15:05.023220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.023062 6001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:15:05.023220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.023073 6001 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-130.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:15:05.023220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.023129 6001 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.059793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.059759 6001 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.059962 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.059859 6001 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.081754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.081693 6001 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal"] Nov 03 16:15:05.082510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.082492 6001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.082593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.082477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"b90d8c1e0bb8232281e68f4371e9fed3","Type":"ContainerStarted","Data":"b6d7e346e25c50380a49b8464c4a39937f7244492738fc7a3b2db5d3f3ac95d9"} Nov 03 16:15:05.082647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.082606 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" event={"ID":"b90d8c1e0bb8232281e68f4371e9fed3","Type":"ContainerStarted","Data":"171471f20dfb24bb22e1a65b7cced2dc04e9c4bd1e51eb6e02207279d4245205"} Nov 03 16:15:05.082712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.082693 6001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.118088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.118002 6001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:15:05.118088 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:05.118061 6001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.118268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.118002 6001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:15:05.118268 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:05.118181 6001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.206492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.206433 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.206492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.206492 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.206704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.206547 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307026 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307085 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307150 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307187 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b90d8c1e0bb8232281e68f4371e9fed3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" (UID: \"b90d8c1e0bb8232281e68f4371e9fed3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.307244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.307192 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45286eece609f171819d9469f4da982c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal\" (UID: \"45286eece609f171819d9469f4da982c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.732928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.732879 6001 apiserver.go:52] "Watching apiserver" Nov 03 16:15:05.801149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.799169 6001 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:15:05.801715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.801681 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qdl9p","openshift-cluster-node-tuning-operator/tuned-lht4j","openshift-dns/dns-default-2dsmp","openshift-monitoring/node-exporter-n9q8w","openshift-network-operator/iptables-alerter-qb7s9","openshift-monitoring/thanos-querier-5765448457-qf2dp","openshift-network-diagnostics/network-check-target-zhgz9","openshift-console/console-7765985f69-vdrnv","openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz","openshift-ingress-canary/ingress-canary-rnjkc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-130.ec2.internal","openshift-monitoring/metrics-server-68fbd467cc-f5zvj","openshift-ovn-kubernetes/ovnkube-node-c84vd","openshift-dns/node-resolver-t6s9c","openshift-insights/insights-runtime-extractor-phsml","openshift-multus/multus-hwdlx","openshift-multus/network-metrics-daemon-qgkpn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r","openshift-console/console-54d7b79479-75lgp","openshift-console/console-597cf7cc48-9wxwj","kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal","kube-system/global-pull-secret-syncer-4qqx2","openshift-monitoring/alertmanager-main-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7","kube-system/konnectivity-agent-h2qrh","openshift-monitoring/monitoring-plugin-77f4f76594-n97fl","openshift-image-registry/node-ca-4tjbw","openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:05.820087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.820048 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.820087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.820067 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.820282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.820102 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.820708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.820678 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:15:05.820820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.820782 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:15:05.821036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821016 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:15:05.821230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821209 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:15:05.821293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821256 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:15:05.821338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821212 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.821522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821502 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.821584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821222 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:15:05.821632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821621 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:15:05.821814 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821799 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:15:05.821886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821836 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:15:05.821886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821804 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:15:05.822010 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.821991 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.822064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.822029 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.822419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.822399 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:15:05.826857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.826838 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:15:05.827003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.826986 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:15:05.827501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.827488 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:15:05.827542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.827525 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:15:05.827651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.827634 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:15:05.827682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.827654 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.827796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.827781 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:15:05.828286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.828270 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:15:05.828338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.828309 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.828479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.828465 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.828964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.828948 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 16:15:05.828964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.828961 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:15:05.829096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829003 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.829096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829023 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:15:05.829184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829100 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.829184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829102 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:15:05.829184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829123 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:15:05.829288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829214 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.829288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829212 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:15:05.829288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829216 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 16:15:05.829382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829298 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.829585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829563 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.829648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829570 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 16:15:05.829648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829603 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:15:05.829758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829745 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 16:15:05.829812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.829767 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:15:05.830013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.830001 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p9flk\"" Nov 03 16:15:05.830065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.830012 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qrl28\"" Nov 03 16:15:05.830233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.830218 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:15:05.831908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.831885 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:15:05.833679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.833664 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:15:05.834339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.834321 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:15:05.834417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.834405 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:15:05.834741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.834721 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:15:05.836078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836064 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:15:05.836326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836313 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.836502 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836488 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 16:15:05.836613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836583 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 16:15:05.836613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836586 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 16:15:05.836613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836600 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.836613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836613 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.836881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836633 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.836881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836640 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 16:15:05.836881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836755 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:15:05.836881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836871 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836920 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836935 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836945 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836972 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836975 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836936 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836988 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836985 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.836996 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ccbfl\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837061 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.837171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837139 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837177 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4ddc504v5k29s\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837213 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4j92k\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837239 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837257 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k6446\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837269 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837279 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837273 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837295 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837300 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e8gqemq6h18bl\"" Nov 03 16:15:05.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.837258 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:15:05.850483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.850457 6001 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:15:05.854465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.854429 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:15:05.863756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.863734 6001 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.885768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.885742 6001 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:15:05.885906 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:05.885798 6001 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-130.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-130.ec2.internal" Nov 03 16:15:05.886790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.886763 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 16:15:05.886790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.886787 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 16:15:05.886973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.886913 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 16:15:05.886973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.886958 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 16:15:05.887070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.886959 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-5zgcl\"" Nov 03 16:15:05.887070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887032 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 16:15:05.887070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887053 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887070 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887095 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887036 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887118 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887114 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887146 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kx5mg\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887118 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887166 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887174 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cpmbj\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887041 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ehkrv9jhbpoh\"" Nov 03 16:15:05.887223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887211 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-drxjc\"" Nov 03 16:15:05.887537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.887241 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 16:15:05.909126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909093 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.909126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.909337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909143 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6928\" (UniqueName: \"kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:15:05.909337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909159 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.909337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909264 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-tmp\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.909337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909266 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.909337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909321 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxqrw\" (UniqueName: \"kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909342 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909367 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909384 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909410 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909438 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909472 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909497 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tphwm\" (UniqueName: \"kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.909588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909555 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.909890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909593 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.909890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909610 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.909890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909701 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.909890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909807 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-daemon-config\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.909890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909819 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909892 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rzx\" (UniqueName: \"kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.910070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909970 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.910070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.909999 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910002 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.910070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910049 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910100 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910171 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910213 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910256 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.910298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910260 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910342 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.910525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910369 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.910525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910477 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.910525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910515 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.910662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910546 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-client-certs\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.910662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910560 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:05.910725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910658 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.910725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910687 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.910725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910712 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.910861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910714 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.910861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910754 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhxv\" (UniqueName: \"kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.910861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910778 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.910861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910804 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb5t\" (UniqueName: \"kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:05.910861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910841 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910868 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2sdp\" (UniqueName: \"kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910910 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/26796971-9faf-4381-8f8b-23db59a37fcc-config-volume\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910932 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910959 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.910988 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911025 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.911103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911054 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911104 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911116 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911155 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911185 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911194 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911215 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911241 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911270 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911297 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911318 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911347 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911362 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911371 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911487 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7gg\" (UniqueName: \"kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911522 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-config\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911533 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911557 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911596 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911624 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87cd7\" (UniqueName: \"kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911651 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911705 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e757f958-921a-4cc9-9735-c3f5594b55d9-cert\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911727 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911848 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911855 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911877 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911891 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911903 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911938 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911964 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.911980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.911987 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912033 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-tls\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912049 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912079 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912108 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912118 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0811b831-8592-4677-aa31-cd58433a3f5c-iptables-alerter-script\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912137 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d16e7255-2933-413f-9114-4b1e2f93d744-ovn-node-metrics-cert\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912221 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912266 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-audit-log\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912298 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912319 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912345 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912374 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912380 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bc21144a-4ab2-4b38-b190-54a63a17c90c-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912391 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-etc-tuned\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912401 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzt6k\" (UniqueName: \"kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912382 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912426 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912466 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912493 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912527 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912521 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjntj\" (UniqueName: \"kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912573 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912573 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912600 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912643 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-grpc-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912684 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912717 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-ovnkube-script-lib\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912716 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.912747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912772 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912801 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912857 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912891 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912888 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912951 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912983 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.912998 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913010 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913061 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6cd40ea-b486-4d7d-a8ec-49d153976a09-tmp-dir\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913069 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913107 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913136 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913161 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913195 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913227 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913291 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913315 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913331 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913340 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913372 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/746b87e4-de05-417f-82ba-53e481c27611-tmp\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913389 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913390 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913432 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913432 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913479 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913516 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913546 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913574 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913621 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913669 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913709 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913746 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj497\" (UniqueName: \"kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913810 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913837 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.913994 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914013 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914054 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914057 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914087 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914105 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914128 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a163d0db-7c08-45c9-bf83-6132d280a72a-serviceca\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:05.914141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914161 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjcx5\" (UniqueName: \"kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914179 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914195 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914219 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914268 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914300 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914332 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914362 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914408 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914438 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-secret-metrics-server-tls\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914436 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914490 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/debced67-f8f8-480e-a1bc-4add3059fd0e-agent-certs\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914542 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914573 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914599 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914670 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914673 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914712 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914740 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914763 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914781 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914785 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914821 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914839 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914849 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914857 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914918 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvp5\" (UniqueName: \"kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914961 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d16e7255-2933-413f-9114-4b1e2f93d744-env-overrides\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.914995 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915020 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915022 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26796971-9faf-4381-8f8b-23db59a37fcc-tmp-dir\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915040 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915067 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915104 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915131 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915162 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915192 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915201 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915202 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915218 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915298 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cni-binary-copy\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915326 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915354 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915359 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-client-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915409 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf95c1a4-4c19-4f20-8014-49afdffd700b-metrics-certs\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915465 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915496 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915524 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915553 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915582 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915610 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/26796971-9faf-4381-8f8b-23db59a37fcc-metrics-tls\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915618 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/debced67-f8f8-480e-a1bc-4add3059fd0e-konnectivity-ca\") pod \"konnectivity-agent-h2qrh\" (UID: \"debced67-f8f8-480e-a1bc-4add3059fd0e\") " pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915611 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915653 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915671 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915712 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nql7\" (UniqueName: \"kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915731 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915748 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915763 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915778 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915777 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915838 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915870 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915900 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915908 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915927 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915955 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.915982 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916008 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916033 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916093 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.916078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916120 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916146 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916294 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916381 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916379 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/deeed625-5069-4da6-8482-2c8ca7c4ff88-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916428 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916465 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916477 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-secret-thanos-querier-tls\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916492 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916520 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-textfile\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916535 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916560 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916595 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916620 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916681 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916799 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916803 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916831 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fde7e318-0d30-4892-81f1-3e4f19c782f5-cni-binary-copy\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916839 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916852 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916897 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916934 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.916975 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917019 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917137 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917145 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917180 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917203 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f18098b-6d38-4cba-85ed-5e110a988df8-data-volume\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917244 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917356 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvqh\" (UniqueName: \"kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917409 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917505 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917557 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng4vr\" (UniqueName: \"kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917589 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-original-pull-secret\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917645 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.917834 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.919080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.918035 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f18098b-6d38-4cba-85ed-5e110a988df8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.920177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920024 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 16:15:05.920177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920037 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.920308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920292 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 16:15:05.920349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920337 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:15:05.920471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920456 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.920517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920468 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 16:15:05.920894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920873 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 16:15:05.920894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920874 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:15:05.920894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920912 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 16:15:05.921339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.920918 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:15:05.921339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.921170 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-metrics-server-audit-profiles\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.921517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.921410 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:15:05.922845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.922822 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.924110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.924082 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.924260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.924237 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.924535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.924513 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.924790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.924772 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.925073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.925054 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.925126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.925078 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/deeed625-5069-4da6-8482-2c8ca7c4ff88-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.926734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.926706 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.926825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.926787 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-metrics-client-ca\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.929563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.929538 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-accelerators-collector-config\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.930194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.930168 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.930281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.930170 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-metrics-client-ca\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.934298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.934242 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 16:15:05.934428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.934368 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 16:15:05.934930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.934910 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:15:05.934930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.934917 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 16:15:05.935712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.935686 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.936232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.936216 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxqrw\" (UniqueName: \"kubernetes.io/projected/bf95c1a4-4c19-4f20-8014-49afdffd700b-kube-api-access-xxqrw\") pod \"network-metrics-daemon-qgkpn\" (UID: \"bf95c1a4-4c19-4f20-8014-49afdffd700b\") " pod="openshift-multus/network-metrics-daemon-qgkpn" Nov 03 16:15:05.936438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.936420 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6928\" (UniqueName: \"kubernetes.io/projected/e757f958-921a-4cc9-9735-c3f5594b55d9-kube-api-access-p6928\") pod \"ingress-canary-rnjkc\" (UID: \"e757f958-921a-4cc9-9735-c3f5594b55d9\") " pod="openshift-ingress-canary/ingress-canary-rnjkc" Nov 03 16:15:05.936499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.936422 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tphwm\" (UniqueName: \"kubernetes.io/projected/0de08296-dbf4-45f3-8dc4-33cbf8de0658-kube-api-access-tphwm\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:05.936530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.936516 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.936706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.936689 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.941680 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.941657 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.941772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.941664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.952135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.952107 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"console-597cf7cc48-9wxwj\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:05.952287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.952267 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"console-54d7b79479-75lgp\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:05.952335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.952267 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nql7\" (UniqueName: \"kubernetes.io/projected/746b87e4-de05-417f-82ba-53e481c27611-kube-api-access-7nql7\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.956868 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzt6k\" (UniqueName: \"kubernetes.io/projected/4f18098b-6d38-4cba-85ed-5e110a988df8-kube-api-access-tzt6k\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.956965 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjntj\" (UniqueName: \"kubernetes.io/projected/caa2b183-e4e8-41fc-85e4-52293e0add64-kube-api-access-jjntj\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957306 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"prometheus-k8s-0\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957317 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rzx\" (UniqueName: \"kubernetes.io/projected/deeed625-5069-4da6-8482-2c8ca7c4ff88-kube-api-access-c9rzx\") pod \"kube-state-metrics-5dc8d7d89b-dg7h7\" (UID: \"deeed625-5069-4da6-8482-2c8ca7c4ff88\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dg7h7" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957382 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjcx5\" (UniqueName: \"kubernetes.io/projected/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kube-api-access-tjcx5\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957385 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvqh\" (UniqueName: \"kubernetes.io/projected/0811b831-8592-4677-aa31-cd58433a3f5c-kube-api-access-phvqh\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:05.957439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957396 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqwz\" (UniqueName: \"kubernetes.io/projected/efef1ee8-f44f-4b0f-9c5d-de4024cfb8da-kube-api-access-mvqwz\") pod \"metrics-server-68fbd467cc-f5zvj\" (UID: \"efef1ee8-f44f-4b0f-9c5d-de4024cfb8da\") " pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:05.957817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957636 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5mm\" (UniqueName: \"kubernetes.io/projected/b125f59a-0236-430f-9c41-57402f877adc-kube-api-access-bj5mm\") pod \"network-check-target-zhgz9\" (UID: \"b125f59a-0236-430f-9c41-57402f877adc\") " pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:15:05.957817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.957695 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7gg\" (UniqueName: \"kubernetes.io/projected/bc21144a-4ab2-4b38-b190-54a63a17c90c-kube-api-access-rp7gg\") pod \"monitoring-plugin-77f4f76594-n97fl\" (UID: \"bc21144a-4ab2-4b38-b190-54a63a17c90c\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:15:05.958182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.958165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"alertmanager-main-0\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:05.958291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.958269 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"console-7765985f69-vdrnv\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:05.958362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.958348 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87cd7\" (UniqueName: \"kubernetes.io/projected/4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54-kube-api-access-87cd7\") pod \"thanos-querier-5765448457-qf2dp\" (UID: \"4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54\") " pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:05.961297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.961279 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng4vr\" (UniqueName: \"kubernetes.io/projected/fde7e318-0d30-4892-81f1-3e4f19c782f5-kube-api-access-ng4vr\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:05.961552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.961534 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvp5\" (UniqueName: \"kubernetes.io/projected/a13a995e-bff6-4420-b8d3-4d2f107c2fa4-kube-api-access-pwvp5\") pod \"openshift-state-metrics-7bd7489957-qj6xz\" (UID: \"a13a995e-bff6-4420-b8d3-4d2f107c2fa4\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qj6xz" Nov 03 16:15:05.961827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.961811 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb5t\" (UniqueName: \"kubernetes.io/projected/a163d0db-7c08-45c9-bf83-6132d280a72a-kube-api-access-vbb5t\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:05.961947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.961933 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2sdp\" (UniqueName: \"kubernetes.io/projected/26796971-9faf-4381-8f8b-23db59a37fcc-kube-api-access-h2sdp\") pod \"dns-default-2dsmp\" (UID: \"26796971-9faf-4381-8f8b-23db59a37fcc\") " pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:05.963638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.963621 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhxv\" (UniqueName: \"kubernetes.io/projected/d16e7255-2933-413f-9114-4b1e2f93d744-kube-api-access-sbhxv\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:05.965329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:05.965310 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj497\" (UniqueName: \"kubernetes.io/projected/f6cd40ea-b486-4d7d-a8ec-49d153976a09-kube-api-access-fj497\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:06.018267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018183 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.018267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018221 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.018267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018238 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018306 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-bin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018358 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018394 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f6cd40ea-b486-4d7d-a8ec-49d153976a09-hosts-file\") pod \"node-resolver-t6s9c\" (UID: \"f6cd40ea-b486-4d7d-a8ec-49d153976a09\") " pod="openshift-dns/node-resolver-t6s9c" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018408 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-node-exporter-wtmp\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018401 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018364 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-lib-modules\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018439 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-netns\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018463 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018494 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f18098b-6d38-4cba-85ed-5e110a988df8-crio-socket\") pod \"insights-runtime-extractor-phsml\" (UID: \"4f18098b-6d38-4cba-85ed-5e110a988df8\") " pod="openshift-insights/insights-runtime-extractor-phsml" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018508 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018539 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018563 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018580 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-etc-kubernetes\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.018599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018602 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-registration-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018589 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018631 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018656 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-socket-dir-parent\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018667 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018673 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018694 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018704 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysconfig\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018715 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018731 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018745 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018753 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-sys-fs\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018763 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018785 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018788 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-system-cni-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018792 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-var-lib-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018809 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018815 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-conf\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018829 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-root\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018818 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/caa2b183-e4e8-41fc-85e4-52293e0add64-sys\") pod \"node-exporter-n9q8w\" (UID: \"caa2b183-e4e8-41fc-85e4-52293e0add64\") " pod="openshift-monitoring/node-exporter-n9q8w" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018844 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-etc-openvswitch\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018832 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018798 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-hostroot\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018889 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018863 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-ovn\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018913 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018945 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018938 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-cnibin\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018951 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-k8s-cni-cncf-io\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018968 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.018985 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019002 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019024 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019050 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-netns\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019028 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-var-lib-kubelet\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019055 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-node-log\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019071 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0811b831-8592-4677-aa31-cd58433a3f5c-host-slash\") pod \"iptables-alerter-qb7s9\" (UID: \"0811b831-8592-4677-aa31-cd58433a3f5c\") " pod="openshift-network-operator/iptables-alerter-qb7s9" Nov 03 16:15:06.019056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019080 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019055 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-cnibin\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019051 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-run-multus-certs\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019136 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019155 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019162 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-log-socket\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019175 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019195 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019208 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-system-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019218 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-kubelet\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019230 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019239 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-slash\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019256 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019261 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-run-systemd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019264 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-bin\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019286 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019316 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019325 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-cni-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019328 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019347 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019359 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a163d0db-7c08-45c9-bf83-6132d280a72a-host\") pod \"node-ca-4tjbw\" (UID: \"a163d0db-7c08-45c9-bf83-6132d280a72a\") " pod="openshift-image-registry/node-ca-4tjbw" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019369 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019385 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-kubernetes\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019390 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019433 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-kubelet-config\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019435 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019490 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019506 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-sysctl-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019514 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019534 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a27eb936-8e3e-4bfb-8c6a-05c184ee89ea-dbus\") pod \"global-pull-secret-syncer-4qqx2\" (UID: \"a27eb936-8e3e-4bfb-8c6a-05c184ee89ea\") " pod="kube-system/global-pull-secret-syncer-4qqx2" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019538 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019564 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019558 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-modprobe-d\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019580 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019584 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0de08296-dbf4-45f3-8dc4-33cbf8de0658-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qdl9p\" (UID: \"0de08296-dbf4-45f3-8dc4-33cbf8de0658\") " pod="openshift-multus/multus-additional-cni-plugins-qdl9p" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019595 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-etc-systemd\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019595 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019622 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019624 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-host\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019636 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-kubelet\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019645 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-systemd-units\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019646 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019668 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019676 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-cni-netd\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019684 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019668 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019741 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019759 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-run\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019765 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-socket-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019769 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019791 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-multus-conf-dir\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019793 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019813 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019821 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d16e7255-2933-413f-9114-4b1e2f93d744-host-run-ovn-kubernetes\") pod \"ovnkube-node-c84vd\" (UID: \"d16e7255-2933-413f-9114-4b1e2f93d744\") " pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019822 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/746b87e4-de05-417f-82ba-53e481c27611-sys\") pod \"tuned-lht4j\" (UID: \"746b87e4-de05-417f-82ba-53e481c27611\") " pod="openshift-cluster-node-tuning-operator/tuned-lht4j" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019833 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019849 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019872 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-device-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019900 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fde7e318-0d30-4892-81f1-3e4f19c782f5-host-var-lib-cni-multus\") pod \"multus-hwdlx\" (UID: \"fde7e318-0d30-4892-81f1-3e4f19c782f5\") " pod="openshift-multus/multus-hwdlx" Nov 03 16:15:06.020266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.019911 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8555d4b-9866-49fa-b635-d9b8e26f8b3b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dsr4r\" (UID: \"a8555d4b-9866-49fa-b635-d9b8e26f8b3b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dsr4r" Nov 03 16:15:06.251705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.251679 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:06.258034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.258011 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:06.262638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.262616 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:06.428255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:06.428178 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:15:07.646329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:07.646288 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2dsmp" Nov 03 16:15:07.926020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:07.925938 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h2qrh" Nov 03 16:15:08.172559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:08.172525 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5765448457-qf2dp" Nov 03 16:15:08.512593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:08.512564 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:08.531217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:08.531191 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:11.170728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:11.170678 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:15:12.974349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:12.974324 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:12.974770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:12.974561 6001 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:15:13.500155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:13.500116 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:14.686207 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:14.686166 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-n97fl" Nov 03 16:15:14.849237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:14.849201 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:18.082618 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:18.082542 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68fbd467cc-f5zvj" Nov 03 16:15:22.249889 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:22.249849 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:15:23.404647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.404615 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:23.405087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.404915 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="alertmanager" containerID="cri-o://e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" gracePeriod=120 Nov 03 16:15:23.405087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.404982 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-metric" containerID="cri-o://ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" gracePeriod=120 Nov 03 16:15:23.405087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.405001 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-web" containerID="cri-o://1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" gracePeriod=120 Nov 03 16:15:23.405087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.405047 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="config-reloader" containerID="cri-o://064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" gracePeriod=120 Nov 03 16:15:23.405306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.405117 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy" containerID="cri-o://c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" gracePeriod=120 Nov 03 16:15:23.405306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.405046 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="prom-label-proxy" containerID="cri-o://84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" gracePeriod=120 Nov 03 16:15:23.505479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.505430 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:15:23.940119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940080 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" exitCode=0 Nov 03 16:15:23.940119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940106 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" exitCode=0 Nov 03 16:15:23.940119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940116 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" exitCode=0 Nov 03 16:15:23.940119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940126 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" exitCode=0 Nov 03 16:15:23.940395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940155 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8"} Nov 03 16:15:23.940395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940219 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91"} Nov 03 16:15:23.940395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940245 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e"} Nov 03 16:15:23.940395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:23.940270 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0"} Nov 03 16:15:24.679678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.679632 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:24.797856 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.797818 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.797875 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.797926 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.797973 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798001 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798022 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798037 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798067 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798094 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798129 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798158 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798196 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.798364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.798265 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") pod \"92928fd5-df80-494a-b99f-1e675a190dea\" (UID: \"92928fd5-df80-494a-b99f-1e675a190dea\") " Nov 03 16:15:24.799225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.799186 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:24.799708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.799676 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:24.800033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.800007 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:24.801076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.801047 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.801962 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.801918 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.802088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.802019 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:24.802611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.802582 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.803545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.803517 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.804122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.804089 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.804235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.804144 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "kube-api-access-s8lz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:24.804235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.804219 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:24.807747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.807720 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.815540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.815514 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config" pod "92928fd5-df80-494a-b99f-1e675a190dea" (UID: "92928fd5-df80-494a-b99f-1e675a190dea"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:24.899822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899789 6001 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-cluster-tls-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.899822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899820 6001 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-metrics-client-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899832 6001 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899843 6001 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899854 6001 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-main-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899864 6001 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899873 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8lz6\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-kube-api-access-s8lz6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899904 6001 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-web-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899913 6001 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-main-db\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899922 6001 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92928fd5-df80-494a-b99f-1e675a190dea-tls-assets\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899931 6001 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/92928fd5-df80-494a-b99f-1e675a190dea-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899940 6001 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92928fd5-df80-494a-b99f-1e675a190dea-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.900020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.899949 6001 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92928fd5-df80-494a-b99f-1e675a190dea-config-out\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:24.946417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946377 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" exitCode=0 Nov 03 16:15:24.946417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946411 6001 generic.go:356] "Generic (PLEG): container finished" podID="92928fd5-df80-494a-b99f-1e675a190dea" containerID="1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" exitCode=0 Nov 03 16:15:24.946654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946460 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0"} Nov 03 16:15:24.946654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946504 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:24.946654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946543 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea"} Nov 03 16:15:24.946654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946590 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"92928fd5-df80-494a-b99f-1e675a190dea","Type":"ContainerDied","Data":"685c83415098532adeaed1332e17fc1f13e1ab94a98cbbd45205dcd6fb73a2ca"} Nov 03 16:15:24.946654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.946616 6001 scope.go:122] "RemoveContainer" containerID="84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" Nov 03 16:15:24.955591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.955571 6001 scope.go:122] "RemoveContainer" containerID="ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" Nov 03 16:15:24.962969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.962950 6001 scope.go:122] "RemoveContainer" containerID="c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" Nov 03 16:15:24.970123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.970104 6001 scope.go:122] "RemoveContainer" containerID="1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" Nov 03 16:15:24.976878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.976861 6001 scope.go:122] "RemoveContainer" containerID="064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" Nov 03 16:15:24.984576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.984562 6001 scope.go:122] "RemoveContainer" containerID="e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" Nov 03 16:15:24.992167 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.992150 6001 scope.go:122] "RemoveContainer" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" Nov 03 16:15:24.999072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.999049 6001 scope.go:122] "RemoveContainer" containerID="84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:24.999381 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8\": container with ID starting with 84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8 not found: ID does not exist" containerID="84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.999423 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8"} err="failed to get container status \"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8\": rpc error: code = NotFound desc = could not find container \"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8\": container with ID starting with 84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8 not found: ID does not exist" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.999502 6001 scope.go:122] "RemoveContainer" containerID="ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:24.999870 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0\": container with ID starting with ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0 not found: ID does not exist" containerID="ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.999901 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0"} err="failed to get container status \"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0\": rpc error: code = NotFound desc = could not find container \"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0\": container with ID starting with ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0 not found: ID does not exist" Nov 03 16:15:25.000080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:24.999924 6001 scope.go:122] "RemoveContainer" containerID="c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" Nov 03 16:15:25.000884 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:25.000184 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91\": container with ID starting with c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91 not found: ID does not exist" containerID="c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" Nov 03 16:15:25.000884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.000211 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91"} err="failed to get container status \"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91\": rpc error: code = NotFound desc = could not find container \"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91\": container with ID starting with c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91 not found: ID does not exist" Nov 03 16:15:25.000884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.000231 6001 scope.go:122] "RemoveContainer" containerID="1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" Nov 03 16:15:25.000884 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:25.000867 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea\": container with ID starting with 1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea not found: ID does not exist" containerID="1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" Nov 03 16:15:25.001060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.000896 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea"} err="failed to get container status \"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea\": rpc error: code = NotFound desc = could not find container \"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea\": container with ID starting with 1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea not found: ID does not exist" Nov 03 16:15:25.001060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.000914 6001 scope.go:122] "RemoveContainer" containerID="064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" Nov 03 16:15:25.001246 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:25.001225 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e\": container with ID starting with 064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e not found: ID does not exist" containerID="064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" Nov 03 16:15:25.001289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.001252 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e"} err="failed to get container status \"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e\": rpc error: code = NotFound desc = could not find container \"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e\": container with ID starting with 064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e not found: ID does not exist" Nov 03 16:15:25.001289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.001272 6001 scope.go:122] "RemoveContainer" containerID="e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" Nov 03 16:15:25.001683 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:25.001660 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0\": container with ID starting with e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0 not found: ID does not exist" containerID="e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" Nov 03 16:15:25.001763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.001713 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0"} err="failed to get container status \"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0\": rpc error: code = NotFound desc = could not find container \"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0\": container with ID starting with e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0 not found: ID does not exist" Nov 03 16:15:25.001763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.001732 6001 scope.go:122] "RemoveContainer" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" Nov 03 16:15:25.002005 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:25.001984 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8\": container with ID starting with 092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8 not found: ID does not exist" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" Nov 03 16:15:25.002092 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002007 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8"} err="failed to get container status \"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8\": rpc error: code = NotFound desc = could not find container \"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8\": container with ID starting with 092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8 not found: ID does not exist" Nov 03 16:15:25.002092 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002022 6001 scope.go:122] "RemoveContainer" containerID="84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8" Nov 03 16:15:25.002327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002302 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8"} err="failed to get container status \"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8\": rpc error: code = NotFound desc = could not find container \"84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8\": container with ID starting with 84cc2888e03329d1b7bdbeca55763bbb6ddb9ecbaa13e24e22fadf3be1297be8 not found: ID does not exist" Nov 03 16:15:25.002387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002327 6001 scope.go:122] "RemoveContainer" containerID="ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0" Nov 03 16:15:25.002621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002601 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0"} err="failed to get container status \"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0\": rpc error: code = NotFound desc = could not find container \"ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0\": container with ID starting with ea32cdedc5e559d4f70bbecc4e6eda1cb4d709a0b9e1ce261be66bdfc424f0c0 not found: ID does not exist" Nov 03 16:15:25.002703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002622 6001 scope.go:122] "RemoveContainer" containerID="c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91" Nov 03 16:15:25.002858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002834 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91"} err="failed to get container status \"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91\": rpc error: code = NotFound desc = could not find container \"c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91\": container with ID starting with c8c9fc1e30cd71033e454e442911ddb613d77c77aafa8bb4332ffc024c434b91 not found: ID does not exist" Nov 03 16:15:25.002908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.002858 6001 scope.go:122] "RemoveContainer" containerID="1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea" Nov 03 16:15:25.003037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003019 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:25.003087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003073 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea"} err="failed to get container status \"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea\": rpc error: code = NotFound desc = could not find container \"1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea\": container with ID starting with 1e9afd1c9a060225077e5fb7930be7b7d7491fdf716d06afe684f32a4436baea not found: ID does not exist" Nov 03 16:15:25.003125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003092 6001 scope.go:122] "RemoveContainer" containerID="064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e" Nov 03 16:15:25.003346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003314 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e"} err="failed to get container status \"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e\": rpc error: code = NotFound desc = could not find container \"064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e\": container with ID starting with 064019a280266df952d91ecb810dd0a10d3b41ff319e7f75ba2ad976c9634d4e not found: ID does not exist" Nov 03 16:15:25.003346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003338 6001 scope.go:122] "RemoveContainer" containerID="e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0" Nov 03 16:15:25.003633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003616 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0"} err="failed to get container status \"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0\": rpc error: code = NotFound desc = could not find container \"e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0\": container with ID starting with e7c621edf77230ca5d2a0c4315e7cd2197c3a0900fae5d4ac7d934a9f5b516e0 not found: ID does not exist" Nov 03 16:15:25.003689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003633 6001 scope.go:122] "RemoveContainer" containerID="092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8" Nov 03 16:15:25.003873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.003849 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8"} err="failed to get container status \"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8\": rpc error: code = NotFound desc = could not find container \"092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8\": container with ID starting with 092fef31ed0831c65a4c5738202ca83c045746fc9bc94a9339e3bf376375f1b8 not found: ID does not exist" Nov 03 16:15:25.006940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.006919 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:25.042888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.042858 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:25.043200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043186 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="config-reloader" Nov 03 16:15:25.043243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043204 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="config-reloader" Nov 03 16:15:25.043243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043219 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy" Nov 03 16:15:25.043243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043224 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy" Nov 03 16:15:25.043243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043242 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-web" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043248 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-web" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043259 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="init-config-reloader" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043264 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="init-config-reloader" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043272 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="alertmanager" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043276 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="alertmanager" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043282 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="prom-label-proxy" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043286 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="prom-label-proxy" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043292 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-metric" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043296 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-metric" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043349 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="init-config-reloader" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043356 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="alertmanager" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043362 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="prom-label-proxy" Nov 03 16:15:25.043366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043368 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-metric" Nov 03 16:15:25.043733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043375 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="config-reloader" Nov 03 16:15:25.043733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043381 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy" Nov 03 16:15:25.043733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.043387 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92928fd5-df80-494a-b99f-1e675a190dea" containerName="kube-rbac-proxy-web" Nov 03 16:15:25.074667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.074603 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.088312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.088282 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:25.100822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.100770 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 16:15:25.100822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.100771 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 16:15:25.101038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.100972 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 16:15:25.102324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102297 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102340 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102429 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102476 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-out\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102575 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102600 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102634 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102667 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7sln\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-kube-api-access-p7sln\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102724 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102741 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-web-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102760 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102778 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.102841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.102822 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.107332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107308 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107340 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107341 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107403 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107413 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107431 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 16:15:25.107480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.107467 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4j92k\"" Nov 03 16:15:25.203674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203686 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203715 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-out\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203744 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203763 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203791 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203805 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7sln\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-kube-api-access-p7sln\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203830 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.203871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.203852 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-web-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204042 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204161 6001 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204185 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.204245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.204240 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.205492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.205318 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.206135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.206031 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7905790a-fc1a-4274-b8a5-cb1d685006e0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.207464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.207424 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.207606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.207576 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.208348 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.208876 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-out\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.209007 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-web-config\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.209130 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.209282 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.209390 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.212256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.210483 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7905790a-fc1a-4274-b8a5-cb1d685006e0-config-volume\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.228354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.228326 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7sln\" (UniqueName: \"kubernetes.io/projected/7905790a-fc1a-4274-b8a5-cb1d685006e0-kube-api-access-p7sln\") pod \"alertmanager-main-0\" (UID: \"7905790a-fc1a-4274-b8a5-cb1d685006e0\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.500479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.500422 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 16:15:25.852336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.852302 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 16:15:25.853270 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:25.853240 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7905790a_fc1a_4274_b8a5_cb1d685006e0.slice/crio-682bff9ac08c2362abf28991bc5c3f9489d454031270fb25250d572dbf820692 WatchSource:0}: Error finding container 682bff9ac08c2362abf28991bc5c3f9489d454031270fb25250d572dbf820692: Status 404 returned error can't find the container with id 682bff9ac08c2362abf28991bc5c3f9489d454031270fb25250d572dbf820692 Nov 03 16:15:25.952469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.952419 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"e3cd7073e5b05633317dc6804e454e21bd02e944c97912b214581abb3adf1d0f"} Nov 03 16:15:25.952613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:25.952516 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"682bff9ac08c2362abf28991bc5c3f9489d454031270fb25250d572dbf820692"} Nov 03 16:15:26.786380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:26.786343 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92928fd5-df80-494a-b99f-1e675a190dea" path="/var/lib/kubelet/pods/92928fd5-df80-494a-b99f-1e675a190dea/volumes" Nov 03 16:15:26.959058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:26.959025 6001 generic.go:356] "Generic (PLEG): container finished" podID="7905790a-fc1a-4274-b8a5-cb1d685006e0" containerID="e3cd7073e5b05633317dc6804e454e21bd02e944c97912b214581abb3adf1d0f" exitCode=0 Nov 03 16:15:26.959394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:26.959108 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerDied","Data":"e3cd7073e5b05633317dc6804e454e21bd02e944c97912b214581abb3adf1d0f"} Nov 03 16:15:27.965979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.965949 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"df3fb51db6dd6b1a2c18b967c0645d4d9ff25a09776be62b20ccd3758b023df1"} Nov 03 16:15:27.966371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.966013 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"f68ef78a7eba799b961264c35432ef5c401b397e1bc4ef9773e5cacf9c9825b6"} Nov 03 16:15:27.966371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.966043 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"de0ae934c63252ff52a1d68b614e3e862eeca8c6925455b7ce60658fef22a974"} Nov 03 16:15:27.966371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.966070 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"5d0c8ee368dfecb9b73d9dd011169220e078d05b6c280abb61e0be317045fe3b"} Nov 03 16:15:27.966371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.966094 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"3ba91f022a35c277d32c54d21d521a4028aed550984c1a48ddda61630218735e"} Nov 03 16:15:27.966371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:27.966119 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7905790a-fc1a-4274-b8a5-cb1d685006e0","Type":"ContainerStarted","Data":"c11e9dcef9d81c34e136492c02f3657b5d11119522246c40b5b24af4fd915884"} Nov 03 16:15:28.035388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.035333 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.02362511 podStartE2EDuration="3.035319257s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.694147ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:28.034906731 +0000 UTC m=+23.368120300" watchObservedRunningTime="2026-11-03 16:15:28.035319257 +0000 UTC m=+23.368532827" Nov 03 16:15:28.075342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075311 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:28.075659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075626 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="prometheus" containerID="cri-o://5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" gracePeriod=600 Nov 03 16:15:28.075802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075663 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy" containerID="cri-o://e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93" gracePeriod=600 Nov 03 16:15:28.075802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075681 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="thanos-sidecar" containerID="cri-o://a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4" gracePeriod=600 Nov 03 16:15:28.075802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075707 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="config-reloader" containerID="cri-o://c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76" gracePeriod=600 Nov 03 16:15:28.075802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075682 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-web" containerID="cri-o://a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89" gracePeriod=600 Nov 03 16:15:28.075802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.075683 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-thanos" containerID="cri-o://89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c" gracePeriod=600 Nov 03 16:15:28.531430 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:28.531377 6001 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 03 16:15:28.531621 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:28.531469 6001 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 03 16:15:28.531621 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:28.531488 6001 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 03 16:15:28.531621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.531505 6001 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Nov 03 16:15:28.973175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973144 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c" exitCode=0 Nov 03 16:15:28.973175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973173 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93" exitCode=0 Nov 03 16:15:28.973175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973180 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4" exitCode=0 Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973186 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76" exitCode=0 Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973194 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" exitCode=0 Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973214 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c"} Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973287 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93"} Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973328 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4"} Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973363 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76"} Nov 03 16:15:28.973612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:28.973398 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f"} Nov 03 16:15:29.340784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.340756 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:29.446880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.446846 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.446895 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.446925 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.446959 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.446996 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447029 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447110 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447150 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447184 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447217 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447253 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447284 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447312 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447343 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447354 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447412 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447468 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447477 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447504 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447535 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447571 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.447760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447640 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") pod \"391a45b7-3643-4fa0-994f-dc590a257672\" (UID: \"391a45b7-3643-4fa0-994f-dc590a257672\") " Nov 03 16:15:29.448134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447857 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.448134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447948 6001 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.448134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447969 6001 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.448134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.447985 6001 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.448660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.448419 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.448660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.448465 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:29.449740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.449714 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.450050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.449922 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.450050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.449953 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.450531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.450488 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:29.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.453894 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:29.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.453908 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.454075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.453990 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.455344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.455320 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.455486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.455436 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.455722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.455692 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.455821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.455760 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.455887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.455868 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.456814 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.456683 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "kube-api-access-f798l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:29.457735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.457709 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:29.465943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.465917 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config" pod "391a45b7-3643-4fa0-994f-dc590a257672" (UID: "391a45b7-3643-4fa0-994f-dc590a257672"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549207 6001 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549238 6001 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549250 6001 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-db\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549260 6001 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549269 6001 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549278 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f798l\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-kube-api-access-f798l\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549287 6001 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/391a45b7-3643-4fa0-994f-dc590a257672-config-out\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549295 6001 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-metrics-client-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549303 6001 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-web-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549311 6001 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-grpc-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549319 6001 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549326 6001 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-configmap-metrics-client-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549335 6001 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549343 6001 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/391a45b7-3643-4fa0-994f-dc590a257672-tls-assets\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549353 6001 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549361 6001 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/391a45b7-3643-4fa0-994f-dc590a257672-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.549588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.549368 6001 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/391a45b7-3643-4fa0-994f-dc590a257672-secret-kube-rbac-proxy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:29.980314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.980280 6001 generic.go:356] "Generic (PLEG): container finished" podID="391a45b7-3643-4fa0-994f-dc590a257672" containerID="a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89" exitCode=0 Nov 03 16:15:29.980825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.980342 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89"} Nov 03 16:15:29.980825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.980358 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:29.980825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.980398 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"391a45b7-3643-4fa0-994f-dc590a257672","Type":"ContainerDied","Data":"9e39ef99d585e45a125546f1a2c0a980ed090faf1de896dbeb5fb433a8ed58e6"} Nov 03 16:15:29.980825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.980430 6001 scope.go:122] "RemoveContainer" containerID="89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c" Nov 03 16:15:29.988652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.988635 6001 scope.go:122] "RemoveContainer" containerID="e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93" Nov 03 16:15:29.998165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:29.998146 6001 scope.go:122] "RemoveContainer" containerID="a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89" Nov 03 16:15:30.004957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.004941 6001 scope.go:122] "RemoveContainer" containerID="a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4" Nov 03 16:15:30.011554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.011537 6001 scope.go:122] "RemoveContainer" containerID="c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76" Nov 03 16:15:30.018111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.018097 6001 scope.go:122] "RemoveContainer" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" Nov 03 16:15:30.025091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.025076 6001 scope.go:122] "RemoveContainer" containerID="e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496" Nov 03 16:15:30.031605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.031591 6001 scope.go:122] "RemoveContainer" containerID="89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c" Nov 03 16:15:30.031852 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.031834 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c\": container with ID starting with 89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c not found: ID does not exist" containerID="89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c" Nov 03 16:15:30.031898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.031861 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c"} err="failed to get container status \"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c\": rpc error: code = NotFound desc = could not find container \"89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c\": container with ID starting with 89ce5dda708ffdfdf76c4e401c6e9421272765674171ad3eca42e2d14cf9783c not found: ID does not exist" Nov 03 16:15:30.031898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.031878 6001 scope.go:122] "RemoveContainer" containerID="e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93" Nov 03 16:15:30.032129 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.032110 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93\": container with ID starting with e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93 not found: ID does not exist" containerID="e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93" Nov 03 16:15:30.032169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032135 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93"} err="failed to get container status \"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93\": rpc error: code = NotFound desc = could not find container \"e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93\": container with ID starting with e7b0cd848dc794beb65d3c17b998c1275269c494e1f42ee3e0680eff7a9bbc93 not found: ID does not exist" Nov 03 16:15:30.032169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032153 6001 scope.go:122] "RemoveContainer" containerID="a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89" Nov 03 16:15:30.032370 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.032356 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89\": container with ID starting with a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89 not found: ID does not exist" containerID="a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89" Nov 03 16:15:30.032409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032375 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89"} err="failed to get container status \"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89\": rpc error: code = NotFound desc = could not find container \"a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89\": container with ID starting with a5b30edca5fe5ea5a9a519b3cc047cfbbf3a0f062e473036abb13be42af3fe89 not found: ID does not exist" Nov 03 16:15:30.032409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032388 6001 scope.go:122] "RemoveContainer" containerID="a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4" Nov 03 16:15:30.032639 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.032617 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4\": container with ID starting with a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4 not found: ID does not exist" containerID="a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4" Nov 03 16:15:30.032681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032642 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4"} err="failed to get container status \"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4\": rpc error: code = NotFound desc = could not find container \"a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4\": container with ID starting with a42d5f79a77137185a9138ef59c63cd0eec5ba1ca4fc1118102ee3a79f5b3ac4 not found: ID does not exist" Nov 03 16:15:30.032681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032654 6001 scope.go:122] "RemoveContainer" containerID="c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76" Nov 03 16:15:30.032870 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.032854 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76\": container with ID starting with c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76 not found: ID does not exist" containerID="c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76" Nov 03 16:15:30.032906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032874 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76"} err="failed to get container status \"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76\": rpc error: code = NotFound desc = could not find container \"c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76\": container with ID starting with c6f41bc351e335eaf43726da0c65df5ed2ea1697c5db5ccbe058a15599c29f76 not found: ID does not exist" Nov 03 16:15:30.032906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.032887 6001 scope.go:122] "RemoveContainer" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" Nov 03 16:15:30.033129 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.033114 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f\": container with ID starting with 5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f not found: ID does not exist" containerID="5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f" Nov 03 16:15:30.033215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.033130 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f"} err="failed to get container status \"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f\": rpc error: code = NotFound desc = could not find container \"5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f\": container with ID starting with 5b86b1a22045e6f7c7b28227dae5733614d594db09cbc139fce19b7b1950682f not found: ID does not exist" Nov 03 16:15:30.033215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.033142 6001 scope.go:122] "RemoveContainer" containerID="e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496" Nov 03 16:15:30.033342 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:30.033327 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496\": container with ID starting with e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496 not found: ID does not exist" containerID="e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496" Nov 03 16:15:30.033380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.033347 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496"} err="failed to get container status \"e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496\": rpc error: code = NotFound desc = could not find container \"e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496\": container with ID starting with e37435f268977655471763d9db462e90284a218310a6517da1ee08d9766d1496 not found: ID does not exist" Nov 03 16:15:30.050600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.050579 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:30.063730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.063704 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:30.089567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089539 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:30.089846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089834 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="init-config-reloader" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089847 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="init-config-reloader" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089858 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089863 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089869 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="thanos-sidecar" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089874 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="thanos-sidecar" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089882 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-thanos" Nov 03 16:15:30.089889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089888 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-thanos" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089896 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-web" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089901 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-web" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089915 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="prometheus" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089919 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="prometheus" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089925 6001 cpu_manager.go:414] "containerMap: removing container" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="config-reloader" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089929 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="config-reloader" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089978 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="init-config-reloader" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089985 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089990 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="thanos-sidecar" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.089996 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-thanos" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.090001 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="kube-rbac-proxy-web" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.090007 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="prometheus" Nov 03 16:15:30.090089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.090014 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="391a45b7-3643-4fa0-994f-dc590a257672" containerName="config-reloader" Nov 03 16:15:30.131714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.131685 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:30.131886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.131789 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.148581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.148553 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kx5mg\"" Nov 03 16:15:30.148728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.148619 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 16:15:30.148787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.148744 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 16:15:30.149076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149062 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149082 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149084 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149092 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149104 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149095 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 16:15:30.149149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149123 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ehkrv9jhbpoh\"" Nov 03 16:15:30.149366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149297 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 16:15:30.149428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149414 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 16:15:30.149505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149485 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 16:15:30.149505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.149489 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 16:15:30.162875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.162845 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 16:15:30.162974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.162907 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 16:15:30.256260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256176 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-web-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256223 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256304 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256354 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256424 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256535 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tprxq\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-kube-api-access-tprxq\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256579 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256627 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256786 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256814 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256869 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.256918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256905 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256940 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.256971 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.257021 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.257063 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.257098 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-config-out\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.257147 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.257178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.257170 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357649 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357689 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357707 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357723 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357859 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357907 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.357943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.357936 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.358674 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.358877 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.358094 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.358970 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359086 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.359140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.358969 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359204 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-config-out\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359254 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359519 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359564 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359559 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-web-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359754 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359799 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.359952 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.360122 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.360168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tprxq\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-kube-api-access-tprxq\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.360603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.360577 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.362588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.362506 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.362914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.362890 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.364950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.363519 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.365887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.365853 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.366002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.365912 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.366491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.366429 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.366688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.366663 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d169bc56-6a2e-4d13-a438-32dbfd59d915-config-out\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.367058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.367031 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-web-config\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.367251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.367205 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.368334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.368305 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.368837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.368812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.368929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.368913 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.369222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.369203 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.369381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.369361 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d169bc56-6a2e-4d13-a438-32dbfd59d915-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.370107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.370088 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d169bc56-6a2e-4d13-a438-32dbfd59d915-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.389883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.389852 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tprxq\" (UniqueName: \"kubernetes.io/projected/d169bc56-6a2e-4d13-a438-32dbfd59d915-kube-api-access-tprxq\") pod \"prometheus-k8s-0\" (UID: \"d169bc56-6a2e-4d13-a438-32dbfd59d915\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.579584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.579491 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:30.786653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.786619 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391a45b7-3643-4fa0-994f-dc590a257672" path="/var/lib/kubelet/pods/391a45b7-3643-4fa0-994f-dc590a257672/volumes" Nov 03 16:15:30.834110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.834009 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-597cf7cc48-9wxwj" podUID="68a16c09-5008-44a0-b655-7b1f12ab2317" containerName="console" containerID="cri-o://3d3a67e21520bbb81e2bf85495575cba013705679c748a2c33f1254998cf74f8" gracePeriod=15 Nov 03 16:15:30.943120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.943091 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 16:15:30.944421 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:30.944395 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd169bc56_6a2e_4d13_a438_32dbfd59d915.slice/crio-670d66dabb3ec8f3bf84088f6c795085c9f991a5221c68e43b7eacd48e335443 WatchSource:0}: Error finding container 670d66dabb3ec8f3bf84088f6c795085c9f991a5221c68e43b7eacd48e335443: Status 404 returned error can't find the container with id 670d66dabb3ec8f3bf84088f6c795085c9f991a5221c68e43b7eacd48e335443 Nov 03 16:15:30.987455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.987399 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"670d66dabb3ec8f3bf84088f6c795085c9f991a5221c68e43b7eacd48e335443"} Nov 03 16:15:30.990233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.990205 6001 generic.go:356] "Generic (PLEG): container finished" podID="68a16c09-5008-44a0-b655-7b1f12ab2317" containerID="3d3a67e21520bbb81e2bf85495575cba013705679c748a2c33f1254998cf74f8" exitCode=0 Nov 03 16:15:30.990388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:30.990271 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597cf7cc48-9wxwj" event={"ID":"68a16c09-5008-44a0-b655-7b1f12ab2317","Type":"ContainerDied","Data":"3d3a67e21520bbb81e2bf85495575cba013705679c748a2c33f1254998cf74f8"} Nov 03 16:15:31.160226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.160197 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:31.274268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274222 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274275 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274306 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274411 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274522 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274586 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274623 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") pod \"68a16c09-5008-44a0-b655-7b1f12ab2317\" (UID: \"68a16c09-5008-44a0-b655-7b1f12ab2317\") " Nov 03 16:15:31.274799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274731 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:31.274945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274834 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:31.274945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274858 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:31.275059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274955 6001 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.275059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274974 6001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.275059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.274990 6001 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68a16c09-5008-44a0-b655-7b1f12ab2317-tmp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.275151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.275088 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:31.275151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.275129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:31.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.276919 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:31.277069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.276958 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "kube-api-access-gqmvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:31.277069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.276963 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert" pod "68a16c09-5008-44a0-b655-7b1f12ab2317" (UID: "68a16c09-5008-44a0-b655-7b1f12ab2317"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:31.375555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.375438 6001 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.375555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.375497 6001 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.375555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.375509 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqmvz\" (UniqueName: \"kubernetes.io/projected/68a16c09-5008-44a0-b655-7b1f12ab2317-kube-api-access-gqmvz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.375555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.375519 6001 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68a16c09-5008-44a0-b655-7b1f12ab2317-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.375555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.375528 6001 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68a16c09-5008-44a0-b655-7b1f12ab2317-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:31.887066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.886978 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7765985f69-vdrnv" podUID="70e84b0c-2265-41de-ad75-1d9329e1dc21" containerName="console" containerID="cri-o://0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc" gracePeriod=15 Nov 03 16:15:31.996067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.996034 6001 generic.go:356] "Generic (PLEG): container finished" podID="d169bc56-6a2e-4d13-a438-32dbfd59d915" containerID="b9a96f9c96f1f59a7d68a3bcef82ccdb98fc422047fc0df0a1cb8d9c8abca3da" exitCode=0 Nov 03 16:15:31.996559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.996122 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerDied","Data":"b9a96f9c96f1f59a7d68a3bcef82ccdb98fc422047fc0df0a1cb8d9c8abca3da"} Nov 03 16:15:31.997647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.997624 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597cf7cc48-9wxwj" event={"ID":"68a16c09-5008-44a0-b655-7b1f12ab2317","Type":"ContainerDied","Data":"437c0c5e594cec728544d21ca1bf414c879c0b34e6cedffadd4e1b49b06b127a"} Nov 03 16:15:31.997647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.997646 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597cf7cc48-9wxwj" Nov 03 16:15:31.997831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:31.997679 6001 scope.go:122] "RemoveContainer" containerID="3d3a67e21520bbb81e2bf85495575cba013705679c748a2c33f1254998cf74f8" Nov 03 16:15:32.109424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.109385 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:15:32.116798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.116749 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597cf7cc48-9wxwj"] Nov 03 16:15:32.196008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.195978 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:32.283896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.283871 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.283976 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284005 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284038 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284076 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284108 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284138 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") pod \"70e84b0c-2265-41de-ad75-1d9329e1dc21\" (UID: \"70e84b0c-2265-41de-ad75-1d9329e1dc21\") " Nov 03 16:15:32.284876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.284852 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:32.285803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.285780 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:32.286067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.286050 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:32.286129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.286104 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:32.290093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.290063 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:32.290359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.290327 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:32.292048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.292020 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v" pod "70e84b0c-2265-41de-ad75-1d9329e1dc21" (UID: "70e84b0c-2265-41de-ad75-1d9329e1dc21"). InnerVolumeSpecName "kube-api-access-9rr6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:32.385521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385496 6001 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385523 6001 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385540 6001 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385551 6001 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70e84b0c-2265-41de-ad75-1d9329e1dc21-tmp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385565 6001 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385579 6001 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/70e84b0c-2265-41de-ad75-1d9329e1dc21-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.385760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.385592 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rr6v\" (UniqueName: \"kubernetes.io/projected/70e84b0c-2265-41de-ad75-1d9329e1dc21-kube-api-access-9rr6v\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:15:32.786715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:32.786678 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68a16c09-5008-44a0-b655-7b1f12ab2317" path="/var/lib/kubelet/pods/68a16c09-5008-44a0-b655-7b1f12ab2317/volumes" Nov 03 16:15:33.003534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.003494 6001 generic.go:356] "Generic (PLEG): container finished" podID="70e84b0c-2265-41de-ad75-1d9329e1dc21" containerID="0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc" exitCode=0 Nov 03 16:15:33.003973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.003557 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7765985f69-vdrnv" Nov 03 16:15:33.003973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.003583 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7765985f69-vdrnv" event={"ID":"70e84b0c-2265-41de-ad75-1d9329e1dc21","Type":"ContainerDied","Data":"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc"} Nov 03 16:15:33.003973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.003641 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7765985f69-vdrnv" event={"ID":"70e84b0c-2265-41de-ad75-1d9329e1dc21","Type":"ContainerDied","Data":"1da53ff5ca617cda1859d868324285f8f0e71c62aed17d11488c7a99d20cf9bf"} Nov 03 16:15:33.003973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.003662 6001 scope.go:122] "RemoveContainer" containerID="0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc" Nov 03 16:15:33.007613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007589 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"d0d0afee18e8bfdca5c105831833318329a658904861149e1576f2e78f7fe2b6"} Nov 03 16:15:33.007701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007661 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"51f8f72f4261525887a70fdd83a3748e2f09b5cf8cddca45eb3992082f5b08e8"} Nov 03 16:15:33.007736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007703 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"d7a6139057be508c183296da82b3d79a4ea1d35072f22ec27d334086fa7e4eaa"} Nov 03 16:15:33.007783 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007744 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"29929f6796c2be61edd18b4ca1405462264ddfcec3cdd3a8947517e811d11818"} Nov 03 16:15:33.007824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007788 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"f01717e38bf42cdc1f6f71c5d65d78f9e471ea91d62abc3bd3f640cdbd522a6d"} Nov 03 16:15:33.007857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.007826 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d169bc56-6a2e-4d13-a438-32dbfd59d915","Type":"ContainerStarted","Data":"e999e328c4191214dabe2d70cd30fe01adb6a35b6dd870057f3c10c1f664b39c"} Nov 03 16:15:33.011990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.011974 6001 scope.go:122] "RemoveContainer" containerID="0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc" Nov 03 16:15:33.012262 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:33.012243 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc\": container with ID starting with 0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc not found: ID does not exist" containerID="0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc" Nov 03 16:15:33.012311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.012271 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc"} err="failed to get container status \"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc\": rpc error: code = NotFound desc = could not find container \"0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc\": container with ID starting with 0643c56a564fd3bfec356564fb63a75e139446df010a448867a3a9f2408963fc not found: ID does not exist" Nov 03 16:15:33.061723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.061637 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:15:33.061982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.061971 6001 cpu_manager.go:414] "containerMap: removing container" podUID="68a16c09-5008-44a0-b655-7b1f12ab2317" containerName="console" Nov 03 16:15:33.062032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.061984 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68a16c09-5008-44a0-b655-7b1f12ab2317" containerName="console" Nov 03 16:15:33.062032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.061995 6001 cpu_manager.go:414] "containerMap: removing container" podUID="70e84b0c-2265-41de-ad75-1d9329e1dc21" containerName="console" Nov 03 16:15:33.062032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.062000 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e84b0c-2265-41de-ad75-1d9329e1dc21" containerName="console" Nov 03 16:15:33.062127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.062067 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68a16c09-5008-44a0-b655-7b1f12ab2317" containerName="console" Nov 03 16:15:33.062127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.062074 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e84b0c-2265-41de-ad75-1d9329e1dc21" containerName="console" Nov 03 16:15:33.064737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.064713 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.087083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.087054 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:15:33.099794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.099745 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.07177552 podStartE2EDuration="3.099731809s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.956289ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:33.094698712 +0000 UTC m=+28.427912281" watchObservedRunningTime="2026-11-03 16:15:33.099731809 +0000 UTC m=+28.432945372" Nov 03 16:15:33.130625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.130588 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:15:33.137996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.137960 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7765985f69-vdrnv"] Nov 03 16:15:33.193628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193592 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnr4d\" (UniqueName: \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.193628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193630 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.193882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193677 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.193882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193736 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.193882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193828 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.193882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193876 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.194090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.193959 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.194090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.194009 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.294867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.294867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294874 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294900 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294949 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.294987 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295019 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295083 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnr4d\" (UniqueName: \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295748 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295822 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295842 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295898 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.295974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.295903 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.297545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.297522 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.297635 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.297563 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.319650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.319565 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnr4d\" (UniqueName: \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\") pod \"console-77648db56f-ctb8z\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.375710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.375672 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:33.673211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:33.673182 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:15:33.674351 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:15:33.674326 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9c9f1_a566_458d_ba12_b9c804640d62.slice/crio-4a361076e4e28ec6676b1369639a127334c1247da732233f482c39c3e2091bc8 WatchSource:0}: Error finding container 4a361076e4e28ec6676b1369639a127334c1247da732233f482c39c3e2091bc8: Status 404 returned error can't find the container with id 4a361076e4e28ec6676b1369639a127334c1247da732233f482c39c3e2091bc8 Nov 03 16:15:34.013359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:34.013319 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77648db56f-ctb8z" event={"ID":"36f9c9f1-a566-458d-ba12-b9c804640d62","Type":"ContainerStarted","Data":"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040"} Nov 03 16:15:34.013833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:34.013382 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77648db56f-ctb8z" event={"ID":"36f9c9f1-a566-458d-ba12-b9c804640d62","Type":"ContainerStarted","Data":"4a361076e4e28ec6676b1369639a127334c1247da732233f482c39c3e2091bc8"} Nov 03 16:15:34.059874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:34.059823 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-77648db56f-ctb8z" podStartSLOduration=1.059809924 podStartE2EDuration="1.059809924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:15:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:15:34.059304055 +0000 UTC m=+29.392517624" watchObservedRunningTime="2026-11-03 16:15:34.059809924 +0000 UTC m=+29.393023484" Nov 03 16:15:34.791796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:34.791763 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e84b0c-2265-41de-ad75-1d9329e1dc21" path="/var/lib/kubelet/pods/70e84b0c-2265-41de-ad75-1d9329e1dc21/volumes" Nov 03 16:15:35.580312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:35.580261 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:36.075674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:36.075642 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c84vd" Nov 03 16:15:41.174080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:41.174048 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zhgz9" Nov 03 16:15:43.381473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:43.379478 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:43.381473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:43.380870 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:43.388573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:43.388548 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:44.059802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:44.059772 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:15:44.165954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:44.165920 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:15:51.207328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:51.207285 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:51.223019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:51.222994 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:52.107723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:15:52.107694 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 16:15:52.254055 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:15:52.254018 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:16:09.191530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.191467 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54d7b79479-75lgp" podUID="46511848-529c-41d9-a9ae-d09fa48ddd3d" containerName="console" containerID="cri-o://5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432" gracePeriod=15 Nov 03 16:16:09.433791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.433764 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:16:09.557812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557779 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557822 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557841 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557877 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557914 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.557947 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558017 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558073 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") pod \"46511848-529c-41d9-a9ae-d09fa48ddd3d\" (UID: \"46511848-529c-41d9-a9ae-d09fa48ddd3d\") " Nov 03 16:16:09.558380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558268 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:09.558380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558348 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:09.558506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558404 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:09.558728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558706 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:16:09.558817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558770 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:09.558873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558852 6001 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.558873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558863 6001 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46511848-529c-41d9-a9ae-d09fa48ddd3d-tmp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.558873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558872 6001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.559024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558881 6001 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.559024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.558889 6001 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/46511848-529c-41d9-a9ae-d09fa48ddd3d-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.560412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.560387 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:16:09.560525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.560463 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "kube-api-access-569mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:16:09.560525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.560518 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert" pod "46511848-529c-41d9-a9ae-d09fa48ddd3d" (UID: "46511848-529c-41d9-a9ae-d09fa48ddd3d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:16:09.642116 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.642079 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-559f9fb54-tnqtx"] Nov 03 16:16:09.642892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.642871 6001 cpu_manager.go:414] "containerMap: removing container" podUID="46511848-529c-41d9-a9ae-d09fa48ddd3d" containerName="console" Nov 03 16:16:09.643009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.642892 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46511848-529c-41d9-a9ae-d09fa48ddd3d" containerName="console" Nov 03 16:16:09.643088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.643075 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46511848-529c-41d9-a9ae-d09fa48ddd3d" containerName="console" Nov 03 16:16:09.646722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.646684 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.659727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.659708 6001 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.659818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.659747 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569mw\" (UniqueName: \"kubernetes.io/projected/46511848-529c-41d9-a9ae-d09fa48ddd3d-kube-api-access-569mw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.659818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.659757 6001 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/46511848-529c-41d9-a9ae-d09fa48ddd3d-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:09.671707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.671683 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559f9fb54-tnqtx"] Nov 03 16:16:09.761146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761107 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-trusted-ca-bundle\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761152 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24b4\" (UniqueName: \"kubernetes.io/projected/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-kube-api-access-d24b4\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761266 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-oauth-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761317 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-service-ca\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761333 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-tmp\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761481 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.761607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.761508 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-oauth-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862084 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-oauth-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862145 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-service-ca\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862167 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-tmp\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862248 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-oauth-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862305 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-trusted-ca-bundle\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.862518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.862335 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24b4\" (UniqueName: \"kubernetes.io/projected/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-kube-api-access-d24b4\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.863027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.863001 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-tmp\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.863148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.863024 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-oauth-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.863148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.863052 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.863642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.863233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-service-ca\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.863642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.863594 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-trusted-ca-bundle\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.864986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.864955 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-serving-cert\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.865075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.864993 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-console-oauth-config\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.887377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.887351 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24b4\" (UniqueName: \"kubernetes.io/projected/74a03a30-b6cf-4f1f-8560-1ab5ca56647e-kube-api-access-d24b4\") pod \"console-559f9fb54-tnqtx\" (UID: \"74a03a30-b6cf-4f1f-8560-1ab5ca56647e\") " pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:09.957369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:09.957330 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:10.165900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.165812 6001 generic.go:356] "Generic (PLEG): container finished" podID="46511848-529c-41d9-a9ae-d09fa48ddd3d" containerID="5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432" exitCode=0 Nov 03 16:16:10.165900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.165877 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54d7b79479-75lgp" Nov 03 16:16:10.165900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.165885 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7b79479-75lgp" event={"ID":"46511848-529c-41d9-a9ae-d09fa48ddd3d","Type":"ContainerDied","Data":"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432"} Nov 03 16:16:10.166124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.165940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54d7b79479-75lgp" event={"ID":"46511848-529c-41d9-a9ae-d09fa48ddd3d","Type":"ContainerDied","Data":"8f62bec151afa46fd093a30bb3805f856f0b4c45f74c5fad7a39b74ad0022d3e"} Nov 03 16:16:10.166124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.165961 6001 scope.go:122] "RemoveContainer" containerID="5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432" Nov 03 16:16:10.174642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.174625 6001 scope.go:122] "RemoveContainer" containerID="5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432" Nov 03 16:16:10.174904 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:16:10.174884 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432\": container with ID starting with 5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432 not found: ID does not exist" containerID="5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432" Nov 03 16:16:10.174973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.174914 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432"} err="failed to get container status \"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432\": rpc error: code = NotFound desc = could not find container \"5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432\": container with ID starting with 5fed8ea1d671665a54976bdb0e2172a09f9f4394fef33b88e265cc8ae6d15432 not found: ID does not exist" Nov 03 16:16:10.218965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.218936 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:16:10.231995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.231965 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54d7b79479-75lgp"] Nov 03 16:16:10.264249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.264221 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559f9fb54-tnqtx"] Nov 03 16:16:10.265707 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:16:10.265679 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a03a30_b6cf_4f1f_8560_1ab5ca56647e.slice/crio-95cc631296225320d66911cebd64235ae222b86fc015522f713289ebb0dd0e34 WatchSource:0}: Error finding container 95cc631296225320d66911cebd64235ae222b86fc015522f713289ebb0dd0e34: Status 404 returned error can't find the container with id 95cc631296225320d66911cebd64235ae222b86fc015522f713289ebb0dd0e34 Nov 03 16:16:10.786407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:10.786369 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46511848-529c-41d9-a9ae-d09fa48ddd3d" path="/var/lib/kubelet/pods/46511848-529c-41d9-a9ae-d09fa48ddd3d/volumes" Nov 03 16:16:11.172594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:11.172509 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559f9fb54-tnqtx" event={"ID":"74a03a30-b6cf-4f1f-8560-1ab5ca56647e","Type":"ContainerStarted","Data":"498c674165e1ba1928053f1f23056e943b78597d6a68d0bae3bef3fb2c7a9190"} Nov 03 16:16:11.172594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:11.172566 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559f9fb54-tnqtx" event={"ID":"74a03a30-b6cf-4f1f-8560-1ab5ca56647e","Type":"ContainerStarted","Data":"95cc631296225320d66911cebd64235ae222b86fc015522f713289ebb0dd0e34"} Nov 03 16:16:11.219055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:11.219004 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-559f9fb54-tnqtx" podStartSLOduration=2.218989564 podStartE2EDuration="2.218989564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:16:11.218361093 +0000 UTC m=+66.551574661" watchObservedRunningTime="2026-11-03 16:16:11.218989564 +0000 UTC m=+66.552203132" Nov 03 16:16:19.957541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:19.957502 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:19.958134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:19.957566 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:19.962403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:19.962383 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:20.207424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:20.207390 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-559f9fb54-tnqtx" Nov 03 16:16:20.376290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:20.376258 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:16:22.249732 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:16:22.249698 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:16:45.395882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.395815 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-77648db56f-ctb8z" podUID="36f9c9f1-a566-458d-ba12-b9c804640d62" containerName="console" containerID="cri-o://db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040" gracePeriod=15 Nov 03 16:16:45.642070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.642040 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:16:45.782087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782047 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782089 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\" (UniqueName: \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782115 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782220 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\" (UniqueName: \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782258 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782295 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\") pod \"36f9c9f1-a566-458d-ba12-b9c804640d62\" (UID: \"36f9c9f1-a566-458d-ba12-b9c804640d62\") " Nov 03 16:16:45.782674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782437 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:16:45.782674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782557 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:45.782674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782585 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:45.782829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782797 6001 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-console-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.782829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782820 6001 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36f9c9f1-a566-458d-ba12-b9c804640d62-tmp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.782927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782834 6001 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.782927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782840 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:45.782927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.782873 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:16:45.784691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.784663 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:16:45.784792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.784689 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "kube-api-access-mnr4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:16:45.784792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.784728 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config" pod "36f9c9f1-a566-458d-ba12-b9c804640d62" (UID: "36f9c9f1-a566-458d-ba12-b9c804640d62"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:16:45.883553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.883494 6001 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-trusted-ca-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.883553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.883545 6001 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-oauth-config\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.883553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.883556 6001 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36f9c9f1-a566-458d-ba12-b9c804640d62-oauth-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.883553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.883564 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnr4d\" (UniqueName: \"kubernetes.io/projected/36f9c9f1-a566-458d-ba12-b9c804640d62-kube-api-access-mnr4d\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:45.883553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:45.883575 6001 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36f9c9f1-a566-458d-ba12-b9c804640d62-console-serving-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:16:46.299543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.299493 6001 generic.go:356] "Generic (PLEG): container finished" podID="36f9c9f1-a566-458d-ba12-b9c804640d62" containerID="db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040" exitCode=0 Nov 03 16:16:46.299543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.299543 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77648db56f-ctb8z" Nov 03 16:16:46.299779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.299556 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77648db56f-ctb8z" event={"ID":"36f9c9f1-a566-458d-ba12-b9c804640d62","Type":"ContainerDied","Data":"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040"} Nov 03 16:16:46.299779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.299615 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77648db56f-ctb8z" event={"ID":"36f9c9f1-a566-458d-ba12-b9c804640d62","Type":"ContainerDied","Data":"4a361076e4e28ec6676b1369639a127334c1247da732233f482c39c3e2091bc8"} Nov 03 16:16:46.299779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.299638 6001 scope.go:122] "RemoveContainer" containerID="db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040" Nov 03 16:16:46.308836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.308819 6001 scope.go:122] "RemoveContainer" containerID="db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040" Nov 03 16:16:46.309127 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:16:46.309109 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040\": container with ID starting with db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040 not found: ID does not exist" containerID="db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040" Nov 03 16:16:46.309175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.309139 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040"} err="failed to get container status \"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040\": rpc error: code = NotFound desc = could not find container \"db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040\": container with ID starting with db9763429d5aafaa1586cf3f628ea09c1bdeded8545d5b1a17a132546ffb9040 not found: ID does not exist" Nov 03 16:16:46.354828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.354795 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:16:46.367064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.367033 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77648db56f-ctb8z"] Nov 03 16:16:46.786295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:16:46.786257 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f9c9f1-a566-458d-ba12-b9c804640d62" path="/var/lib/kubelet/pods/36f9c9f1-a566-458d-ba12-b9c804640d62/volumes" Nov 03 16:16:52.250258 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:16:52.250216 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:17:22.249243 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:17:22.249199 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:17:52.249703 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:17:52.249661 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:18:22.248336 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:18:22.248254 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:18:36.803087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.803051 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh"] Nov 03 16:18:36.803503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.803400 6001 cpu_manager.go:414] "containerMap: removing container" podUID="36f9c9f1-a566-458d-ba12-b9c804640d62" containerName="console" Nov 03 16:18:36.803503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.803410 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f9c9f1-a566-458d-ba12-b9c804640d62" containerName="console" Nov 03 16:18:36.803503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.803483 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f9c9f1-a566-458d-ba12-b9c804640d62" containerName="console" Nov 03 16:18:36.806649 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.806628 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:36.834996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.834964 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 16:18:36.835246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.834989 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 16:18:36.835246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.835107 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-flw2l\"" Nov 03 16:18:36.836059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.836038 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh"] Nov 03 16:18:36.941951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.941915 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:36.942158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.941971 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qqc\" (UniqueName: \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:36.942158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:36.942112 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.042879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.042833 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.043078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.042896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qqc\" (UniqueName: \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.043078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.042961 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.043332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.043315 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.043380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.043313 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.069415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.069330 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qqc\" (UniqueName: \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.123070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.123028 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:37.421794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.421767 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh"] Nov 03 16:18:37.423382 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:18:37.423349 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66174279_8bcc_4e44_9a80_92fff6273381.slice/crio-64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129 WatchSource:0}: Error finding container 64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129: Status 404 returned error can't find the container with id 64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129 Nov 03 16:18:37.425709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.425685 6001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:18:37.710863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:37.710784 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" event={"ID":"66174279-8bcc-4e44-9a80-92fff6273381","Type":"ContainerStarted","Data":"64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129"} Nov 03 16:18:43.736858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:43.736815 6001 generic.go:356] "Generic (PLEG): container finished" podID="66174279-8bcc-4e44-9a80-92fff6273381" containerID="60cdc03b77144d2f0f10009f628261503b9bb77f343657638b3ef30ee98893cc" exitCode=0 Nov 03 16:18:43.737342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:43.736902 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" event={"ID":"66174279-8bcc-4e44-9a80-92fff6273381","Type":"ContainerDied","Data":"60cdc03b77144d2f0f10009f628261503b9bb77f343657638b3ef30ee98893cc"} Nov 03 16:18:44.742994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:44.742957 6001 generic.go:356] "Generic (PLEG): container finished" podID="66174279-8bcc-4e44-9a80-92fff6273381" containerID="da552d10e16b09779e6ec2305ba2852a79b1015fd9a49d1944e0908ea66ffcbd" exitCode=0 Nov 03 16:18:44.743393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:44.743017 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" event={"ID":"66174279-8bcc-4e44-9a80-92fff6273381","Type":"ContainerDied","Data":"da552d10e16b09779e6ec2305ba2852a79b1015fd9a49d1944e0908ea66ffcbd"} Nov 03 16:18:51.776207 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:51.776170 6001 generic.go:356] "Generic (PLEG): container finished" podID="66174279-8bcc-4e44-9a80-92fff6273381" containerID="6ab94ac62fab90171e312ac05bb1ec33c5b9229ffd3df13fe6b3f47d4de48157" exitCode=0 Nov 03 16:18:51.776740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:51.776237 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" event={"ID":"66174279-8bcc-4e44-9a80-92fff6273381","Type":"ContainerDied","Data":"6ab94ac62fab90171e312ac05bb1ec33c5b9229ffd3df13fe6b3f47d4de48157"} Nov 03 16:18:52.249102 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:18:52.249065 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:18:52.912233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.912206 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:18:52.985209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.985179 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\") pod \"66174279-8bcc-4e44-9a80-92fff6273381\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " Nov 03 16:18:52.985362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.985224 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\") pod \"66174279-8bcc-4e44-9a80-92fff6273381\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " Nov 03 16:18:52.985362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.985284 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\" (UniqueName: \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\") pod \"66174279-8bcc-4e44-9a80-92fff6273381\" (UID: \"66174279-8bcc-4e44-9a80-92fff6273381\") " Nov 03 16:18:52.985873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.985835 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle" pod "66174279-8bcc-4e44-9a80-92fff6273381" (UID: "66174279-8bcc-4e44-9a80-92fff6273381"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:18:52.988021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.987998 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc" pod "66174279-8bcc-4e44-9a80-92fff6273381" (UID: "66174279-8bcc-4e44-9a80-92fff6273381"). InnerVolumeSpecName "kube-api-access-76qqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:18:52.990341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:52.990317 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util" pod "66174279-8bcc-4e44-9a80-92fff6273381" (UID: "66174279-8bcc-4e44-9a80-92fff6273381"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:18:53.086007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.085897 6001 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-bundle\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:18:53.086007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.085943 6001 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/66174279-8bcc-4e44-9a80-92fff6273381-util\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:18:53.086007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.085958 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qqc\" (UniqueName: \"kubernetes.io/projected/66174279-8bcc-4e44-9a80-92fff6273381-kube-api-access-76qqc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:18:53.785318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.785282 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" event={"ID":"66174279-8bcc-4e44-9a80-92fff6273381","Type":"ContainerDied","Data":"64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129"} Nov 03 16:18:53.785514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.785327 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64041021d475f902fa77aa82cb2e71545f9ad3d6fe55861d7c8d17e2a76c6129" Nov 03 16:18:53.785514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:18:53.785325 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71hdhnh" Nov 03 16:19:00.233802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.233766 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f"] Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234095 6001 cpu_manager.go:414] "containerMap: removing container" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="util" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234107 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="util" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234136 6001 cpu_manager.go:414] "containerMap: removing container" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="pull" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234141 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="pull" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234148 6001 cpu_manager.go:414] "containerMap: removing container" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="extract" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234152 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="extract" Nov 03 16:19:00.235252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.234218 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="66174279-8bcc-4e44-9a80-92fff6273381" containerName="extract" Nov 03 16:19:00.236553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.236522 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.272056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.272027 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 16:19:00.275521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.275500 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 16:19:00.285120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.285089 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 16:19:00.295512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.295474 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f"] Nov 03 16:19:00.340126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340085 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.340529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340496 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgp8t\" (UniqueName: \"kubernetes.io/projected/31d31064-6530-4b3c-a4f0-4b286c70a91a-kube-api-access-jgp8t\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.340628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340549 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.340696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340671 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.340756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340720 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.340813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.340760 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.359342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.359297 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 16:19:00.359604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.359585 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 16:19:00.359715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.359696 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k6tn8\"" Nov 03 16:19:00.442293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442246 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442300 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442328 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442400 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442474 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgp8t\" (UniqueName: \"kubernetes.io/projected/31d31064-6530-4b3c-a4f0-4b286c70a91a-kube-api-access-jgp8t\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442509 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.442846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.442736 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.444922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.444884 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d31064-6530-4b3c-a4f0-4b286c70a91a-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.445062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.445039 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.445467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.445422 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.445515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.445438 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/31d31064-6530-4b3c-a4f0-4b286c70a91a-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.470211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.470182 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgp8t\" (UniqueName: \"kubernetes.io/projected/31d31064-6530-4b3c-a4f0-4b286c70a91a-kube-api-access-jgp8t\") pod \"opendatahub-operator-controller-manager-698cf769bf-x485f\" (UID: \"31d31064-6530-4b3c-a4f0-4b286c70a91a\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.549259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.549170 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:00.903222 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:00.903193 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d31064_6530_4b3c_a4f0_4b286c70a91a.slice/crio-b669004c52330211dd9e51b6b62de7e2fe2e58f909ae975da3130809699f6a0b WatchSource:0}: Error finding container b669004c52330211dd9e51b6b62de7e2fe2e58f909ae975da3130809699f6a0b: Status 404 returned error can't find the container with id b669004c52330211dd9e51b6b62de7e2fe2e58f909ae975da3130809699f6a0b Nov 03 16:19:00.905601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:00.905582 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f"] Nov 03 16:19:01.823390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:01.823350 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" event={"ID":"31d31064-6530-4b3c-a4f0-4b286c70a91a","Type":"ContainerStarted","Data":"b669004c52330211dd9e51b6b62de7e2fe2e58f909ae975da3130809699f6a0b"} Nov 03 16:19:04.838042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:04.838006 6001 generic.go:356] "Generic (PLEG): container finished" podID="31d31064-6530-4b3c-a4f0-4b286c70a91a" containerID="6c4517c08e1811aa526f49c2a66ae83daa865bb2346605e57c0470465db0c697" exitCode=0 Nov 03 16:19:04.838497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:04.838090 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" event={"ID":"31d31064-6530-4b3c-a4f0-4b286c70a91a","Type":"ContainerDied","Data":"6c4517c08e1811aa526f49c2a66ae83daa865bb2346605e57c0470465db0c697"} Nov 03 16:19:05.843944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:05.843902 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" event={"ID":"31d31064-6530-4b3c-a4f0-4b286c70a91a","Type":"ContainerStarted","Data":"25796fa079eb1b1191be13a260a13b8f816e3b606c6be6ba586102e6bad72949"} Nov 03 16:19:05.844406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:05.844018 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:05.944088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:05.944029 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" podStartSLOduration=2.065438741 podStartE2EDuration="5.944014605s" totalImagesPullingTime="3.428446753s" totalInitContainerRuntime="450.129111ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:05.930368097 +0000 UTC m=+241.263581677" watchObservedRunningTime="2026-11-03 16:19:05.944014605 +0000 UTC m=+241.277228170" Nov 03 16:19:14.640922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.640882 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk"] Nov 03 16:19:14.645190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.645164 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.662912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.662885 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 16:19:14.662912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.662897 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:19:14.676121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.676089 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7b9j4\"" Nov 03 16:19:14.677121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.677102 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk"] Nov 03 16:19:14.768847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.768807 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f490b655-ecc4-4399-8628-a88ae240a6ca-tmp\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.769020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.768864 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzxl\" (UniqueName: \"kubernetes.io/projected/f490b655-ecc4-4399-8628-a88ae240a6ca-kube-api-access-hbzxl\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.869599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.869563 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f490b655-ecc4-4399-8628-a88ae240a6ca-tmp\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.869768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.869615 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzxl\" (UniqueName: \"kubernetes.io/projected/f490b655-ecc4-4399-8628-a88ae240a6ca-kube-api-access-hbzxl\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.869994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.869974 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f490b655-ecc4-4399-8628-a88ae240a6ca-tmp\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.897995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.897928 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzxl\" (UniqueName: \"kubernetes.io/projected/f490b655-ecc4-4399-8628-a88ae240a6ca-kube-api-access-hbzxl\") pod \"jobset-operator-674cd59dc5-hqqjk\" (UID: \"f490b655-ecc4-4399-8628-a88ae240a6ca\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:14.966759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:14.966721 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" Nov 03 16:19:15.273460 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:15.273414 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk"] Nov 03 16:19:15.274800 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:15.274773 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf490b655_ecc4_4399_8628_a88ae240a6ca.slice/crio-ef7d2600e5a1f360eed4660902d27d4da1ff081db698f0d111f7e9a8a37b79e4 WatchSource:0}: Error finding container ef7d2600e5a1f360eed4660902d27d4da1ff081db698f0d111f7e9a8a37b79e4: Status 404 returned error can't find the container with id ef7d2600e5a1f360eed4660902d27d4da1ff081db698f0d111f7e9a8a37b79e4 Nov 03 16:19:15.888196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:15.888153 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" event={"ID":"f490b655-ecc4-4399-8628-a88ae240a6ca","Type":"ContainerStarted","Data":"ef7d2600e5a1f360eed4660902d27d4da1ff081db698f0d111f7e9a8a37b79e4"} Nov 03 16:19:16.852990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:16.852954 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-x485f" Nov 03 16:19:17.900087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:17.900052 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" event={"ID":"f490b655-ecc4-4399-8628-a88ae240a6ca","Type":"ContainerStarted","Data":"039ca9811d7700d88228013745f2217750e2da8f1e652b613e5eda6bc71865f1"} Nov 03 16:19:17.944902 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:17.944840 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hqqjk" podStartSLOduration=1.4601054119999999 podStartE2EDuration="3.944827213s" totalImagesPullingTime="2.484721801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:17.944102789 +0000 UTC m=+253.277316358" watchObservedRunningTime="2026-11-03 16:19:17.944827213 +0000 UTC m=+253.278040779" Nov 03 16:19:22.249197 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:19:22.249159 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:19:34.126576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.126538 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-286sk"] Nov 03 16:19:34.129666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.129647 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.172715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.172688 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hh5nb\"" Nov 03 16:19:34.179707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.179680 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 16:19:34.203348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.203319 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-286sk"] Nov 03 16:19:34.244684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.244648 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.244905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.244738 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nkgm\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-kube-api-access-7nkgm\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.263720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.263686 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 16:19:34.346111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.346073 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.346291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.346150 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nkgm\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-kube-api-access-7nkgm\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.385067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.384997 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.410439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.410399 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nkgm\" (UniqueName: \"kubernetes.io/projected/045a4032-eeeb-46f7-a7f5-ebbbdc01558a-kube-api-access-7nkgm\") pod \"cert-manager-webhook-5dbbbf75cc-286sk\" (UID: \"045a4032-eeeb-46f7-a7f5-ebbbdc01558a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.440917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.440877 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:34.796590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.796555 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-286sk"] Nov 03 16:19:34.796972 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:34.796949 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod045a4032_eeeb_46f7_a7f5_ebbbdc01558a.slice/crio-4d9891fa98b834d7dd3034f00b336f2344cb6ca8430e6f8f533fe07f60695b6d WatchSource:0}: Error finding container 4d9891fa98b834d7dd3034f00b336f2344cb6ca8430e6f8f533fe07f60695b6d: Status 404 returned error can't find the container with id 4d9891fa98b834d7dd3034f00b336f2344cb6ca8430e6f8f533fe07f60695b6d Nov 03 16:19:34.971611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:34.971572 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" event={"ID":"045a4032-eeeb-46f7-a7f5-ebbbdc01558a","Type":"ContainerStarted","Data":"4d9891fa98b834d7dd3034f00b336f2344cb6ca8430e6f8f533fe07f60695b6d"} Nov 03 16:19:38.992070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:38.992029 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" event={"ID":"045a4032-eeeb-46f7-a7f5-ebbbdc01558a","Type":"ContainerStarted","Data":"0c0aae23b33f3346e2fc0df8f2bfc75b6c7ebfa3124d4c0e3b33f265d0c72540"} Nov 03 16:19:38.992496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:38.992108 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:39.036055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:39.035995 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" podStartSLOduration=1.868440076 podStartE2EDuration="5.03597646s" totalImagesPullingTime="3.167536384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:39.035556097 +0000 UTC m=+274.368769666" watchObservedRunningTime="2026-11-03 16:19:39.03597646 +0000 UTC m=+274.369190030" Nov 03 16:19:41.473740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.473705 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c"] Nov 03 16:19:41.477087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.477064 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.502978 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.502947 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rt9w8\"" Nov 03 16:19:41.507400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.507369 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c"] Nov 03 16:19:41.618891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.618848 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5d9\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-kube-api-access-9r5d9\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.619068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.618920 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.720466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.720401 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5d9\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-kube-api-access-9r5d9\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.720644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.720509 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.749763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.749682 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.749897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.749794 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5d9\" (UniqueName: \"kubernetes.io/projected/f647a7f2-34fd-4e9a-aae1-e425f94aa08c-kube-api-access-9r5d9\") pod \"cert-manager-cainjector-5c57bbd4fc-wmd4c\" (UID: \"f647a7f2-34fd-4e9a-aae1-e425f94aa08c\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:41.788605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:41.788566 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" Nov 03 16:19:42.086099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.086059 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c"] Nov 03 16:19:42.087179 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:42.087149 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf647a7f2_34fd_4e9a_aae1_e425f94aa08c.slice/crio-31ec6a583c9d1bdd6b723a37ef272a2a7894b730259ede7c3c63db952a7826c7 WatchSource:0}: Error finding container 31ec6a583c9d1bdd6b723a37ef272a2a7894b730259ede7c3c63db952a7826c7: Status 404 returned error can't find the container with id 31ec6a583c9d1bdd6b723a37ef272a2a7894b730259ede7c3c63db952a7826c7 Nov 03 16:19:42.796843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.796809 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-kgglr"] Nov 03 16:19:42.800186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.800162 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.827558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.827519 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:19:42.827558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.827536 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:19:42.831911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.831879 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601ebbab-626e-4679-a41e-437cdf951a1c-tmp\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.832079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.831931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cms8\" (UniqueName: \"kubernetes.io/projected/601ebbab-626e-4679-a41e-437cdf951a1c-kube-api-access-5cms8\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.832079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.832034 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ebbab-626e-4679-a41e-437cdf951a1c-tls-certs\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.834660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.834634 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 16:19:42.834878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.834860 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8rbg9\"" Nov 03 16:19:42.834945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.834891 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 16:19:42.837532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.837507 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-kgglr"] Nov 03 16:19:42.932950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.932907 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ebbab-626e-4679-a41e-437cdf951a1c-tls-certs\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.933126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.932969 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601ebbab-626e-4679-a41e-437cdf951a1c-tmp\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.933126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.933013 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cms8\" (UniqueName: \"kubernetes.io/projected/601ebbab-626e-4679-a41e-437cdf951a1c-kube-api-access-5cms8\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.935418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.935387 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/601ebbab-626e-4679-a41e-437cdf951a1c-tmp\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.935637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.935619 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/601ebbab-626e-4679-a41e-437cdf951a1c-tls-certs\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:42.960953 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:42.960916 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cms8\" (UniqueName: \"kubernetes.io/projected/601ebbab-626e-4679-a41e-437cdf951a1c-kube-api-access-5cms8\") pod \"kube-auth-proxy-645cdd4567-kgglr\" (UID: \"601ebbab-626e-4679-a41e-437cdf951a1c\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:43.016112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:43.016075 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" event={"ID":"f647a7f2-34fd-4e9a-aae1-e425f94aa08c","Type":"ContainerStarted","Data":"e0a298e26bd1bc85817f1d3dd12cb5129cd31ab5eff19786d509080eb7e86906"} Nov 03 16:19:43.016112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:43.016124 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" event={"ID":"f647a7f2-34fd-4e9a-aae1-e425f94aa08c","Type":"ContainerStarted","Data":"31ec6a583c9d1bdd6b723a37ef272a2a7894b730259ede7c3c63db952a7826c7"} Nov 03 16:19:43.091703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:43.091599 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wmd4c" podStartSLOduration=2.091583067 podStartE2EDuration="2.091583067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:43.091425634 +0000 UTC m=+278.424639203" watchObservedRunningTime="2026-11-03 16:19:43.091583067 +0000 UTC m=+278.424796635" Nov 03 16:19:43.113290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:43.113259 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" Nov 03 16:19:43.425269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:43.425238 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-kgglr"] Nov 03 16:19:43.426512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:43.426474 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601ebbab_626e_4679_a41e_437cdf951a1c.slice/crio-889019aa7a700b491e54a55c77e3fa458aec4ca536878e2ced0236c5bdb7fb54 WatchSource:0}: Error finding container 889019aa7a700b491e54a55c77e3fa458aec4ca536878e2ced0236c5bdb7fb54: Status 404 returned error can't find the container with id 889019aa7a700b491e54a55c77e3fa458aec4ca536878e2ced0236c5bdb7fb54 Nov 03 16:19:44.021519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:44.021474 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" event={"ID":"601ebbab-626e-4679-a41e-437cdf951a1c","Type":"ContainerStarted","Data":"889019aa7a700b491e54a55c77e3fa458aec4ca536878e2ced0236c5bdb7fb54"} Nov 03 16:19:44.998920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:44.998887 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" Nov 03 16:19:45.123997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.123959 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-wz2ht"] Nov 03 16:19:45.132711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.132680 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.148419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.148390 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wz2ht"] Nov 03 16:19:45.151551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.151513 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-k6kl7\"" Nov 03 16:19:45.156070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.156031 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-bound-sa-token\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.156246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.156161 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxpj\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-kube-api-access-bcxpj\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.257338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.257239 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-bound-sa-token\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.257338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.257313 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxpj\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-kube-api-access-bcxpj\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.291566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.291521 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-bound-sa-token\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.291727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.291617 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxpj\" (UniqueName: \"kubernetes.io/projected/5e3afe37-c3a3-4d1d-a547-81b027e347cc-kube-api-access-bcxpj\") pod \"cert-manager-597b95cb84-wz2ht\" (UID: \"5e3afe37-c3a3-4d1d-a547-81b027e347cc\") " pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.448165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.448125 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-wz2ht" Nov 03 16:19:45.762171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:45.762136 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-wz2ht"] Nov 03 16:19:45.763829 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:19:45.763798 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e3afe37_c3a3_4d1d_a547_81b027e347cc.slice/crio-283859f7ebb8dca64bcf22a2636c5af56db440a76e6f66581fcfea93e3b3bd0f WatchSource:0}: Error finding container 283859f7ebb8dca64bcf22a2636c5af56db440a76e6f66581fcfea93e3b3bd0f: Status 404 returned error can't find the container with id 283859f7ebb8dca64bcf22a2636c5af56db440a76e6f66581fcfea93e3b3bd0f Nov 03 16:19:46.030745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:46.030653 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wz2ht" event={"ID":"5e3afe37-c3a3-4d1d-a547-81b027e347cc","Type":"ContainerStarted","Data":"256b5b884ddfcd112d1a62a98286eb723a726f562ef57d6840ca9895f0b089e8"} Nov 03 16:19:46.030745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:46.030703 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-wz2ht" event={"ID":"5e3afe37-c3a3-4d1d-a547-81b027e347cc","Type":"ContainerStarted","Data":"283859f7ebb8dca64bcf22a2636c5af56db440a76e6f66581fcfea93e3b3bd0f"} Nov 03 16:19:46.032221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:46.032187 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" event={"ID":"601ebbab-626e-4679-a41e-437cdf951a1c","Type":"ContainerStarted","Data":"527dd1e7e24affe66ff92b9757e61976d7b20e8736b041719bbbe17567714a61"} Nov 03 16:19:46.080751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:46.080700 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-wz2ht" podStartSLOduration=1.080686347 podStartE2EDuration="1.080686347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:46.078706602 +0000 UTC m=+281.411920171" watchObservedRunningTime="2026-11-03 16:19:46.080686347 +0000 UTC m=+281.413899912" Nov 03 16:19:46.129565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:19:46.129514 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-645cdd4567-kgglr" podStartSLOduration=1.622297424 podStartE2EDuration="4.129498931s" totalImagesPullingTime="2.507201507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:46.128796617 +0000 UTC m=+281.462010188" watchObservedRunningTime="2026-11-03 16:19:46.129498931 +0000 UTC m=+281.462712497" Nov 03 16:19:52.249147 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:19:52.249108 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:20:04.823293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:04.823263 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:20:04.823906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:04.823544 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:20:08.419898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.419862 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq"] Nov 03 16:20:08.426774 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.426741 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.447138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.447104 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq"] Nov 03 16:20:08.458529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.458496 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 16:20:08.458529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.458510 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 16:20:08.463005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.462987 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 16:20:08.463116 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.463004 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-tvh2z\"" Nov 03 16:20:08.567977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.567946 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-cert\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.568164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.568085 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-metrics-certs\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.568164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.568114 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlrl\" (UniqueName: \"kubernetes.io/projected/a08d8018-95da-4a91-bcb8-f09f9f4285a0-kube-api-access-jqlrl\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.568245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.568215 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a08d8018-95da-4a91-bcb8-f09f9f4285a0-manager-config\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.669362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.669316 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-metrics-certs\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.669362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.669362 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlrl\" (UniqueName: \"kubernetes.io/projected/a08d8018-95da-4a91-bcb8-f09f9f4285a0-kube-api-access-jqlrl\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.669664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.669386 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a08d8018-95da-4a91-bcb8-f09f9f4285a0-manager-config\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.669664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.669412 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-cert\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.670278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.670212 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a08d8018-95da-4a91-bcb8-f09f9f4285a0-manager-config\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.672252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.672228 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-cert\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.672372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.672304 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08d8018-95da-4a91-bcb8-f09f9f4285a0-metrics-certs\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.694675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.694641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlrl\" (UniqueName: \"kubernetes.io/projected/a08d8018-95da-4a91-bcb8-f09f9f4285a0-kube-api-access-jqlrl\") pod \"jobset-controller-manager-5f4989f9cc-8b4nq\" (UID: \"a08d8018-95da-4a91-bcb8-f09f9f4285a0\") " pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:08.738404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:08.738367 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:09.047678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:09.047652 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq"] Nov 03 16:20:09.054838 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:20:09.054809 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08d8018_95da_4a91_bcb8_f09f9f4285a0.slice/crio-acb484f9c90e495b0794845ee79fb548c2a68b840ee9e8b473afee1a2a3e9460 WatchSource:0}: Error finding container acb484f9c90e495b0794845ee79fb548c2a68b840ee9e8b473afee1a2a3e9460: Status 404 returned error can't find the container with id acb484f9c90e495b0794845ee79fb548c2a68b840ee9e8b473afee1a2a3e9460 Nov 03 16:20:09.129794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:09.129758 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" event={"ID":"a08d8018-95da-4a91-bcb8-f09f9f4285a0","Type":"ContainerStarted","Data":"acb484f9c90e495b0794845ee79fb548c2a68b840ee9e8b473afee1a2a3e9460"} Nov 03 16:20:12.146704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:12.146609 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" event={"ID":"a08d8018-95da-4a91-bcb8-f09f9f4285a0","Type":"ContainerStarted","Data":"90d27c016312cad832e0508b31633581e91fc750f4b3bda9b5bf07422135bf3a"} Nov 03 16:20:12.147133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:12.146761 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:12.193347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:12.193266 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" podStartSLOduration=1.443140455 podStartE2EDuration="4.193244665s" totalImagesPullingTime="2.75010421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:12.193073519 +0000 UTC m=+307.526287089" watchObservedRunningTime="2026-11-03 16:20:12.193244665 +0000 UTC m=+307.526458234" Nov 03 16:20:22.250079 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:20:22.250039 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:20:23.155823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:20:23.155789 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5f4989f9cc-8b4nq" Nov 03 16:20:52.251353 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:20:52.251313 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:21:22.250366 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:21:22.250282 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:21:52.249848 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:21:52.249800 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:22:22.250087 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:22:22.250017 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:22:52.250132 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:22:52.250091 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:23:22.249178 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:23:22.249137 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:23:52.250159 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:23:52.250120 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:24:22.250257 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:24:22.250171 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:24:52.250340 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:24:52.250302 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:25:04.853971 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:25:04.853937 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:25:04.854749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:25:04.854721 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:25:22.248817 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:25:22.248778 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:25:52.250753 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:25:52.250662 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:26:22.249873 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:26:22.249831 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:26:52.248842 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:26:52.248803 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:27:22.249600 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:27:22.249520 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:27:52.249687 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:27:52.249642 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:28:22.249613 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:28:22.249568 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:28:52.249696 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:28:52.249617 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:29:22.249413 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:29:22.249371 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:29:52.248266 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:29:52.248216 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:30:04.883677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:04.883648 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:30:04.886195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:04.886173 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:30:21.393177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.392980 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8"] Nov 03 16:30:21.397113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.397089 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.417077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.417043 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8"] Nov 03 16:30:21.430943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.430912 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pj2d7\"" Nov 03 16:30:21.431117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.431085 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 16:30:21.444234 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.444193 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 16:30:21.444234 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.444206 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 16:30:21.444234 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.444237 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 16:30:21.482500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.482468 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb8bbc2c-4a30-45a4-a981-63996002b0c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.482673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.482550 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb8bbc2c-4a30-45a4-a981-63996002b0c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.482673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.482606 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9r4\" (UniqueName: \"kubernetes.io/projected/cb8bbc2c-4a30-45a4-a981-63996002b0c4-kube-api-access-rn9r4\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.583210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.583168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb8bbc2c-4a30-45a4-a981-63996002b0c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.583402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.583232 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb8bbc2c-4a30-45a4-a981-63996002b0c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.583402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.583272 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9r4\" (UniqueName: \"kubernetes.io/projected/cb8bbc2c-4a30-45a4-a981-63996002b0c4-kube-api-access-rn9r4\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.583870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.583849 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cb8bbc2c-4a30-45a4-a981-63996002b0c4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.585846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.585812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cb8bbc2c-4a30-45a4-a981-63996002b0c4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.607566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.607531 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9r4\" (UniqueName: \"kubernetes.io/projected/cb8bbc2c-4a30-45a4-a981-63996002b0c4-kube-api-access-rn9r4\") pod \"kuadrant-console-plugin-66c4fb575d-zd2c8\" (UID: \"cb8bbc2c-4a30-45a4-a981-63996002b0c4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:21.709331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:21.709249 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" Nov 03 16:30:22.003401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.003368 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8"] Nov 03 16:30:22.005479 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:30:22.005433 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb8bbc2c_4a30_45a4_a981_63996002b0c4.slice/crio-ac65b470615a60e6cc33c2ec8299722932e8799166ec6facd5c49a151d784e40 WatchSource:0}: Error finding container ac65b470615a60e6cc33c2ec8299722932e8799166ec6facd5c49a151d784e40: Status 404 returned error can't find the container with id ac65b470615a60e6cc33c2ec8299722932e8799166ec6facd5c49a151d784e40 Nov 03 16:30:22.007261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.007242 6001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:30:22.250771 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:30:22.250732 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:30:22.809977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.809920 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" event={"ID":"cb8bbc2c-4a30-45a4-a981-63996002b0c4","Type":"ContainerStarted","Data":"ac65b470615a60e6cc33c2ec8299722932e8799166ec6facd5c49a151d784e40"} Nov 03 16:30:22.966356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.964805 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:30:22.968504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.968477 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:30:22.985690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.985614 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvhsq\"" Nov 03 16:30:22.986786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.986759 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:30:22.996174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:22.996115 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mlx\" (UniqueName: \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\") pod \"authorino-56c6c9ddd5-qx6g6\" (UID: \"a83b0081-3922-4811-bdf6-0f37e01d8a47\") " pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:30:23.097600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:23.097504 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mlx\" (UniqueName: \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\") pod \"authorino-56c6c9ddd5-qx6g6\" (UID: \"a83b0081-3922-4811-bdf6-0f37e01d8a47\") " pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:30:23.122327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:23.122293 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mlx\" (UniqueName: \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\") pod \"authorino-56c6c9ddd5-qx6g6\" (UID: \"a83b0081-3922-4811-bdf6-0f37e01d8a47\") " pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:30:23.283777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:23.283736 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:30:23.587587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:23.587551 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:30:23.589774 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:30:23.589742 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda83b0081_3922_4811_bdf6_0f37e01d8a47.slice/crio-506657a6809cf756e776b2210538832cd4c7295932d645bb5d9220e57b8b6019 WatchSource:0}: Error finding container 506657a6809cf756e776b2210538832cd4c7295932d645bb5d9220e57b8b6019: Status 404 returned error can't find the container with id 506657a6809cf756e776b2210538832cd4c7295932d645bb5d9220e57b8b6019 Nov 03 16:30:23.817406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:23.817359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" event={"ID":"a83b0081-3922-4811-bdf6-0f37e01d8a47","Type":"ContainerStarted","Data":"506657a6809cf756e776b2210538832cd4c7295932d645bb5d9220e57b8b6019"} Nov 03 16:30:26.835975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:26.835933 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" event={"ID":"a83b0081-3922-4811-bdf6-0f37e01d8a47","Type":"ContainerStarted","Data":"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55"} Nov 03 16:30:26.878400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:26.878341 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" podStartSLOduration=2.186242147 podStartE2EDuration="4.878325981s" totalImagesPullingTime="2.692083834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:26.877521609 +0000 UTC m=+922.210735177" watchObservedRunningTime="2026-11-03 16:30:26.878325981 +0000 UTC m=+922.211539549" Nov 03 16:30:27.841655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:27.841611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" event={"ID":"cb8bbc2c-4a30-45a4-a981-63996002b0c4","Type":"ContainerStarted","Data":"229be96cf0f37ca2cc84c3ab0d9c270e466ebc3a6e813ade17055963ea851f5d"} Nov 03 16:30:27.884152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:30:27.884092 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-zd2c8" podStartSLOduration=1.994216136 podStartE2EDuration="6.884075101s" totalImagesPullingTime="4.889858965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:27.88349718 +0000 UTC m=+923.216710748" watchObservedRunningTime="2026-11-03 16:30:27.884075101 +0000 UTC m=+923.217288670" Nov 03 16:30:52.250079 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:30:52.250040 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:31:22.250029 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:31:22.249985 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:31:40.934212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.934174 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v6pms"] Nov 03 16:31:40.937637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.937620 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:40.958070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.958042 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v6pms"] Nov 03 16:31:40.969324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.969301 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 16:31:40.991778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.991744 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v6pms"] Nov 03 16:31:40.992028 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:31:40.992005 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r7lbp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-v6pms" podUID="892ec617-fe16-44df-aedb-5194a51820a6" Nov 03 16:31:40.993754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.993732 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lbp\" (UniqueName: \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:40.993870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:40.993853 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.094864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.094826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lbp\" (UniqueName: \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.095043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.094896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.097267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.097247 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.119101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.119069 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lbp\" (UniqueName: \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\") pod \"authorino-6889cbd8c4-v6pms\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.162322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.162291 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.169570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.169541 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:41.296164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.296135 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\" (UniqueName: \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\") pod \"892ec617-fe16-44df-aedb-5194a51820a6\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " Nov 03 16:31:41.296346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.296221 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\" (UniqueName: \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\") pod \"892ec617-fe16-44df-aedb-5194a51820a6\" (UID: \"892ec617-fe16-44df-aedb-5194a51820a6\") " Nov 03 16:31:41.298215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.298191 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert" pod "892ec617-fe16-44df-aedb-5194a51820a6" (UID: "892ec617-fe16-44df-aedb-5194a51820a6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:41.298215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.298206 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp" pod "892ec617-fe16-44df-aedb-5194a51820a6" (UID: "892ec617-fe16-44df-aedb-5194a51820a6"). InnerVolumeSpecName "kube-api-access-r7lbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:41.396994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.396953 6001 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/892ec617-fe16-44df-aedb-5194a51820a6-tls-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:31:41.396994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:41.396989 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lbp\" (UniqueName: \"kubernetes.io/projected/892ec617-fe16-44df-aedb-5194a51820a6-kube-api-access-r7lbp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:31:42.166513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.166480 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-v6pms" Nov 03 16:31:42.245748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.245714 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v6pms"] Nov 03 16:31:42.249848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.249819 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-v6pms"] Nov 03 16:31:42.798472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.798163 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892ec617-fe16-44df-aedb-5194a51820a6" path="/var/lib/kubelet/pods/892ec617-fe16-44df-aedb-5194a51820a6/volumes" Nov 03 16:31:42.904195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.904159 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:31:42.904386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:42.904344 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" podUID="a83b0081-3922-4811-bdf6-0f37e01d8a47" containerName="authorino" containerID="cri-o://021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55" gracePeriod=30 Nov 03 16:31:43.150898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.150864 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:31:43.173061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.173026 6001 generic.go:356] "Generic (PLEG): container finished" podID="a83b0081-3922-4811-bdf6-0f37e01d8a47" containerID="021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55" exitCode=0 Nov 03 16:31:43.173540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.173058 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" event={"ID":"a83b0081-3922-4811-bdf6-0f37e01d8a47","Type":"ContainerDied","Data":"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55"} Nov 03 16:31:43.173540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.173086 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" Nov 03 16:31:43.173540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.173105 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qx6g6" event={"ID":"a83b0081-3922-4811-bdf6-0f37e01d8a47","Type":"ContainerDied","Data":"506657a6809cf756e776b2210538832cd4c7295932d645bb5d9220e57b8b6019"} Nov 03 16:31:43.173540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.173134 6001 scope.go:122] "RemoveContainer" containerID="021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55" Nov 03 16:31:43.182872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.182845 6001 scope.go:122] "RemoveContainer" containerID="021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55" Nov 03 16:31:43.183173 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:31:43.183151 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55\": container with ID starting with 021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55 not found: ID does not exist" containerID="021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55" Nov 03 16:31:43.183244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.183189 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55"} err="failed to get container status \"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55\": rpc error: code = NotFound desc = could not find container \"021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55\": container with ID starting with 021a29e617c8822bd007b6ccec5fdeaef4880f1d694313f1b9cb1a985b999d55 not found: ID does not exist" Nov 03 16:31:43.214005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.213970 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\" (UniqueName: \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\") pod \"a83b0081-3922-4811-bdf6-0f37e01d8a47\" (UID: \"a83b0081-3922-4811-bdf6-0f37e01d8a47\") " Nov 03 16:31:43.216320 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.216295 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx" pod "a83b0081-3922-4811-bdf6-0f37e01d8a47" (UID: "a83b0081-3922-4811-bdf6-0f37e01d8a47"). InnerVolumeSpecName "kube-api-access-m4mlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:43.314600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.314558 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mlx\" (UniqueName: \"kubernetes.io/projected/a83b0081-3922-4811-bdf6-0f37e01d8a47-kube-api-access-m4mlx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:31:43.519071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.519041 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:31:43.522967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:43.522943 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qx6g6"] Nov 03 16:31:44.786892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:31:44.786854 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83b0081-3922-4811-bdf6-0f37e01d8a47" path="/var/lib/kubelet/pods/a83b0081-3922-4811-bdf6-0f37e01d8a47/volumes" Nov 03 16:31:52.248292 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:31:52.248212 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:32:08.513560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.513525 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:32:08.513968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.513941 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a83b0081-3922-4811-bdf6-0f37e01d8a47" containerName="authorino" Nov 03 16:32:08.513968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.513956 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83b0081-3922-4811-bdf6-0f37e01d8a47" containerName="authorino" Nov 03 16:32:08.514105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.514025 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83b0081-3922-4811-bdf6-0f37e01d8a47" containerName="authorino" Nov 03 16:32:08.518492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.518471 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529222 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529264 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsmn7\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529299 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529369 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529491 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529529 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529588 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529616 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.529663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.529631 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.531625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.531604 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:32:08.535077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.535061 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.543938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.543910 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9tjmt\"" Nov 03 16:32:08.544163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.544134 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 16:32:08.545315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.545286 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:32:08.566797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.566767 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:32:08.630363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630326 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630376 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630398 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630424 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630551 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630595 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630620 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630669 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630697 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsmn7\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630727 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630752 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630792 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630822 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630845 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630866 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630869 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.630967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.630963 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.631407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631016 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.631407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.631407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631196 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmclj\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.631407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631288 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.631407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631332 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.631608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.631551 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.633046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.633025 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.633144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.633052 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.655423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.655394 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.655594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.655576 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsmn7\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mk2z5\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.732060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732025 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732070 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732097 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732125 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732176 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732203 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmclj\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732301 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732329 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732507 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732699 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.732909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.732889 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.733084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.733066 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.734482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.734462 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.734628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.734608 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.756528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.756500 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.756670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.756653 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmclj\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jn8\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:08.832737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.832644 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:08.849938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:08.849910 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:09.145036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:09.144952 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:32:09.152356 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:09.152326 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da63c2_ca5a_4a9f_b177_54a511c9a790.slice/crio-14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956 WatchSource:0}: Error finding container 14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956: Status 404 returned error can't find the container with id 14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956 Nov 03 16:32:09.169803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:09.169773 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:32:09.170820 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:09.170795 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ee91dd_8787_42b8_b981_94782d875b3d.slice/crio-44ff582510692af0fb24652345d07cafad79437adf202d04d2ca187c3f3810d8 WatchSource:0}: Error finding container 44ff582510692af0fb24652345d07cafad79437adf202d04d2ca187c3f3810d8: Status 404 returned error can't find the container with id 44ff582510692af0fb24652345d07cafad79437adf202d04d2ca187c3f3810d8 Nov 03 16:32:09.283263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:09.283222 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" event={"ID":"81ee91dd-8787-42b8-b981-94782d875b3d","Type":"ContainerStarted","Data":"44ff582510692af0fb24652345d07cafad79437adf202d04d2ca187c3f3810d8"} Nov 03 16:32:09.284562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:09.284534 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" event={"ID":"89da63c2-ca5a-4a9f-b177-54a511c9a790","Type":"ContainerStarted","Data":"14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956"} Nov 03 16:32:12.309484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.309431 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:12.309933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.309521 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:12.309933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.309556 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:12.321015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.320985 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:12.321095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.321048 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:12.321095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:12.321079 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:13.309389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.309346 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" event={"ID":"89da63c2-ca5a-4a9f-b177-54a511c9a790","Type":"ContainerStarted","Data":"8b817491a87b3f3ea06f721c7753fa0e5060d4f4c37ed0dc8df8df3ecf34c099"} Nov 03 16:32:13.310980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.310952 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" event={"ID":"81ee91dd-8787-42b8-b981-94782d875b3d","Type":"ContainerStarted","Data":"c1cdfe84c524a535e17dc4fcc818b471a87a6b67f9d4a8dfb980f2403b6b5eb1"} Nov 03 16:32:13.377468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.376600 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podStartSLOduration=2.222338759 podStartE2EDuration="5.376577976s" totalImagesPullingTime="3.154239217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:13.374952076 +0000 UTC m=+1028.708165672" watchObservedRunningTime="2026-11-03 16:32:13.376577976 +0000 UTC m=+1028.709791546" Nov 03 16:32:13.425517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.425457 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podStartSLOduration=2.277949881 podStartE2EDuration="5.42542602s" totalImagesPullingTime="3.147476139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:13.424726992 +0000 UTC m=+1028.757940562" watchObservedRunningTime="2026-11-03 16:32:13.42542602 +0000 UTC m=+1028.758639588" Nov 03 16:32:13.833189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.833140 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:13.834747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.834716 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 16:32:13.834884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.834771 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 16:32:13.863470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.854399 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:13.863470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.854596 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 16:32:13.863470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:13.854645 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 16:32:14.833508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:14.833431 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 16:32:14.833917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:14.833521 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 16:32:14.853470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:14.851905 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 16:32:14.853470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:14.851990 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 16:32:15.833987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:15.833955 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 16:32:15.834379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:15.834007 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 16:32:15.856592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:15.856554 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 16:32:15.856857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:15.856828 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 16:32:16.833484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:16.833096 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 16:32:16.833484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:16.833147 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 16:32:16.867821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:16.867689 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 16:32:16.867821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:16.867764 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 16:32:17.833112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:17.833075 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Nov 03 16:32:17.833290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:17.833129 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Nov 03 16:32:17.853478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:17.853280 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Nov 03 16:32:17.853478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:17.853334 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Nov 03 16:32:18.837815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.837778 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:18.838213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.838084 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:18.838939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.838916 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:32:18.854883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.854859 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:18.855143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.855126 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:18.855849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:18.855833 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:32:22.084534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.084500 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 16:32:22.088140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.088116 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.106177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.106146 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 16:32:22.120188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.120156 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 16:32:22.152609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.152576 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 16:32:22.155986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.155966 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.174632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.174597 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qp7nb\"" Nov 03 16:32:22.175815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.175790 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 16:32:22.186214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.186181 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4n5\" (UniqueName: \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.186472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.186254 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.186472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.186377 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgj6k\" (UniqueName: \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.186472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.186426 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.254124 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:32:22.254088 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:32:22.287709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.287672 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgj6k\" (UniqueName: \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.287709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.287719 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.287964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.287777 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4n5\" (UniqueName: \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.287964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.287860 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.288611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.288587 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.289015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.288980 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.312337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.312301 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgj6k\" (UniqueName: \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\") pod \"payload-pre-processing-7b98bb678c-gblz5\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.312522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.312340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4n5\" (UniqueName: \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\") pod \"payload-processing-55fbbbf9dc-jkphl\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.401291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.401190 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:22.468614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.468579 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:22.700897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.700869 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 16:32:22.702907 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:22.702875 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e3af6e_bac6_4c92_b299_f8c669b9a8d9.slice/crio-50c97516da6c3f9b92bafba617eae9c1b7d93613eaf62d229549a4f710b3d6ea WatchSource:0}: Error finding container 50c97516da6c3f9b92bafba617eae9c1b7d93613eaf62d229549a4f710b3d6ea: Status 404 returned error can't find the container with id 50c97516da6c3f9b92bafba617eae9c1b7d93613eaf62d229549a4f710b3d6ea Nov 03 16:32:22.768571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:22.768545 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 16:32:22.769962 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:22.769934 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ceab1b_3778_4b26_a436_d59dbbe65177.slice/crio-0fc2fcd8652df32fee17124bc81defedebf05a4d35d76b6b27a2bc9d064780f0 WatchSource:0}: Error finding container 0fc2fcd8652df32fee17124bc81defedebf05a4d35d76b6b27a2bc9d064780f0: Status 404 returned error can't find the container with id 0fc2fcd8652df32fee17124bc81defedebf05a4d35d76b6b27a2bc9d064780f0 Nov 03 16:32:23.374777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.374722 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" event={"ID":"f4ceab1b-3778-4b26-a436-d59dbbe65177","Type":"ContainerStarted","Data":"0fc2fcd8652df32fee17124bc81defedebf05a4d35d76b6b27a2bc9d064780f0"} Nov 03 16:32:23.378632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.378580 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" event={"ID":"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9","Type":"ContainerStarted","Data":"50c97516da6c3f9b92bafba617eae9c1b7d93613eaf62d229549a4f710b3d6ea"} Nov 03 16:32:23.594233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.594189 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5"] Nov 03 16:32:23.598728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.598695 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.623323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.623288 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5"] Nov 03 16:32:23.703229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703119 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9732c01-e962-45c8-a46c-c83c88a65a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703209 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703418 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703539 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703580 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703622 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703648 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.703700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.703669 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfvw\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-kube-api-access-5tfvw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.804996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.804945 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805006 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805034 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfvw\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-kube-api-access-5tfvw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805084 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9732c01-e962-45c8-a46c-c83c88a65a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805165 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805225 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805272 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805302 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.805507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.805435 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.806122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.806094 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.806240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.806214 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.806321 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.806302 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.806544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.806511 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9732c01-e962-45c8-a46c-c83c88a65a4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.808663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.808641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.808894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.808875 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.830422 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.830357 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.830822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.830794 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfvw\" (UniqueName: \"kubernetes.io/projected/c9732c01-e962-45c8-a46c-c83c88a65a4e-kube-api-access-5tfvw\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-fmxr5\" (UID: \"c9732c01-e962-45c8-a46c-c83c88a65a4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:23.919466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:23.919398 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:24.248907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.248865 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5"] Nov 03 16:32:24.249955 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:24.249899 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9732c01_e962_45c8_a46c_c83c88a65a4e.slice/crio-69ccd71f25aa88bc510e83a12495d3fca769c09abe70987fe72344cf906cfb02 WatchSource:0}: Error finding container 69ccd71f25aa88bc510e83a12495d3fca769c09abe70987fe72344cf906cfb02: Status 404 returned error can't find the container with id 69ccd71f25aa88bc510e83a12495d3fca769c09abe70987fe72344cf906cfb02 Nov 03 16:32:24.254409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.254374 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:24.254571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.254479 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:24.254571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.254532 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:24.385739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.385628 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" event={"ID":"c9732c01-e962-45c8-a46c-c83c88a65a4e","Type":"ContainerStarted","Data":"1e5315301988ca249ed30248cb2c0ea1cb09bede403441a9cf8fbfc05bb17de0"} Nov 03 16:32:24.385739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.385699 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" event={"ID":"c9732c01-e962-45c8-a46c-c83c88a65a4e","Type":"ContainerStarted","Data":"69ccd71f25aa88bc510e83a12495d3fca769c09abe70987fe72344cf906cfb02"} Nov 03 16:32:24.439876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.439303 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podStartSLOduration=1.439286248 podStartE2EDuration="1.439286248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:24.438619436 +0000 UTC m=+1039.771833005" watchObservedRunningTime="2026-11-03 16:32:24.439286248 +0000 UTC m=+1039.772499818" Nov 03 16:32:24.919578 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:24.919534 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:25.391694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:25.391663 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" event={"ID":"f4ceab1b-3778-4b26-a436-d59dbbe65177","Type":"ContainerStarted","Data":"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50"} Nov 03 16:32:25.392146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:25.391841 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:25.435112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:25.435052 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" podStartSLOduration=1.014723103 podStartE2EDuration="3.435036529s" totalImagesPullingTime="2.420313426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:25.434581677 +0000 UTC m=+1040.767795247" watchObservedRunningTime="2026-11-03 16:32:25.435036529 +0000 UTC m=+1040.768250097" Nov 03 16:32:25.922031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:25.921984 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:25.922244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:25.922050 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:32:26.400403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:26.400362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" event={"ID":"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9","Type":"ContainerStarted","Data":"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228"} Nov 03 16:32:26.400908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:26.400630 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:26.461469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:26.443700 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" podStartSLOduration=1.824303128 podStartE2EDuration="4.443680168s" totalImagesPullingTime="2.61937704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:26.443353675 +0000 UTC m=+1041.776567246" watchObservedRunningTime="2026-11-03 16:32:26.443680168 +0000 UTC m=+1041.776893738" Nov 03 16:32:26.920075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:26.920030 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:26.920289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:26.920103 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:32:27.920732 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:27.920695 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:27.921160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:27.920757 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:32:28.920833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:28.920779 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:28.921241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:28.920861 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:32:29.920075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:29.920036 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:29.920270 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:29.920090 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 16:32:30.925038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:30.925002 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:30.925454 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:30.925295 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:30.926103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:30.926080 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" Nov 03 16:32:36.403878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:36.403842 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 16:32:37.408819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:37.408786 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 16:32:38.622660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.622623 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:32:38.627827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.627801 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.654703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.654659 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:32:38.769998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.769954 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770018 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvws\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770093 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770125 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770244 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770281 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770369 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770393 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.770493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.770418 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871223 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871309 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871337 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871386 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871414 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871496 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871539 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871757 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871818 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvws\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871827 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.871907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.871878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.872224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.872202 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.872308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.872263 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.872308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.872271 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.874104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.874040 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.874487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.874462 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.896020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.895986 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvws\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.896162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.896060 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-d66mm\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:38.940859 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:38.940823 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:39.251313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.251286 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:32:39.252549 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:39.252511 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c1e119_1ffa_47fc_8023_b1d33122dbd5.slice/crio-c8baa4e5a7798bcc2d53444aa1ed76600dd616d253c5e2bf1caf89833012a266 WatchSource:0}: Error finding container c8baa4e5a7798bcc2d53444aa1ed76600dd616d253c5e2bf1caf89833012a266: Status 404 returned error can't find the container with id c8baa4e5a7798bcc2d53444aa1ed76600dd616d253c5e2bf1caf89833012a266 Nov 03 16:32:39.254893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.254863 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:39.254970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.254929 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:39.254970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.254959 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:32:39.462589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.462551 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" event={"ID":"35c1e119-1ffa-47fc-8023-b1d33122dbd5","Type":"ContainerStarted","Data":"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705"} Nov 03 16:32:39.462765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.462611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" event={"ID":"35c1e119-1ffa-47fc-8023-b1d33122dbd5","Type":"ContainerStarted","Data":"c8baa4e5a7798bcc2d53444aa1ed76600dd616d253c5e2bf1caf89833012a266"} Nov 03 16:32:39.512281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.512176 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podStartSLOduration=1.512158884 podStartE2EDuration="1.512158884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:39.510914667 +0000 UTC m=+1054.844128237" watchObservedRunningTime="2026-11-03 16:32:39.512158884 +0000 UTC m=+1054.845372453" Nov 03 16:32:39.941238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:39.941145 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:40.943377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:40.943332 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:40.943876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:40.943397 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:32:41.941389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:41.941353 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:41.941606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:41.941410 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:32:42.942088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:42.942052 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:42.942512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:42.942110 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:32:43.941836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:43.941799 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:43.942126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:43.941856 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:32:44.941554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:44.941511 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:44.941730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:44.941569 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Nov 03 16:32:45.945370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:45.945334 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:45.945869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:45.945680 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:45.946468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:45.946431 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:32:52.250488 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:32:52.250434 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:32:56.801102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.801069 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8"] Nov 03 16:32:56.809515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.809492 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.826827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.826784 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-znvz5\"" Nov 03 16:32:56.828292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.828269 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8"] Nov 03 16:32:56.839581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.839555 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 16:32:56.839714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.839696 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 16:32:56.839778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.839732 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 16:32:56.858005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.857977 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.858139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.858026 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.858255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.858225 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.858365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.858344 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7sx\" (UniqueName: \"kubernetes.io/projected/e52d3d63-9c13-44bd-beca-7370089217c4-kube-api-access-tp7sx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.858469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.858429 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.858563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.858542 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52d3d63-9c13-44bd-beca-7370089217c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.959757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.959716 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.959945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.959850 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.959945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.959910 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.959956 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7sx\" (UniqueName: \"kubernetes.io/projected/e52d3d63-9c13-44bd-beca-7370089217c4-kube-api-access-tp7sx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.959990 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.960007 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52d3d63-9c13-44bd-beca-7370089217c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.960120 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.960204 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.960341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.960323 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.962236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.962208 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e52d3d63-9c13-44bd-beca-7370089217c4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:56.962521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:56.962502 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e52d3d63-9c13-44bd-beca-7370089217c4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:57.000428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:57.000377 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7sx\" (UniqueName: \"kubernetes.io/projected/e52d3d63-9c13-44bd-beca-7370089217c4-kube-api-access-tp7sx\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gf9p8\" (UID: \"e52d3d63-9c13-44bd-beca-7370089217c4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:57.124332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:57.124249 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:32:57.420046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:57.420022 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8"] Nov 03 16:32:57.421367 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:32:57.421342 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52d3d63_9c13_44bd_beca_7370089217c4.slice/crio-5e9da0d8518400818a32cba300b868ded8884b2516f2501bce2fe31948900175 WatchSource:0}: Error finding container 5e9da0d8518400818a32cba300b868ded8884b2516f2501bce2fe31948900175: Status 404 returned error can't find the container with id 5e9da0d8518400818a32cba300b868ded8884b2516f2501bce2fe31948900175 Nov 03 16:32:57.554356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:57.554319 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" event={"ID":"e52d3d63-9c13-44bd-beca-7370089217c4","Type":"ContainerStarted","Data":"5e9da0d8518400818a32cba300b868ded8884b2516f2501bce2fe31948900175"} Nov 03 16:32:59.568941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:59.568908 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" event={"ID":"e52d3d63-9c13-44bd-beca-7370089217c4","Type":"ContainerStarted","Data":"1e4618e3fd399c0ee79c4fef04d1c3c8d0943502a48242877fc323a73081d671"} Nov 03 16:32:59.613982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:32:59.613927 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" podStartSLOduration=1.891128135 podStartE2EDuration="3.613913385s" totalImagesPullingTime="1.72278525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:59.613683456 +0000 UTC m=+1074.946897066" watchObservedRunningTime="2026-11-03 16:32:59.613913385 +0000 UTC m=+1074.947126953" Nov 03 16:33:00.124624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.124589 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:33:00.295093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.295056 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24"] Nov 03 16:33:00.300701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.300669 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.323968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.323936 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24"] Nov 03 16:33:00.330227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.330196 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 16:33:00.397478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397346 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.397478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.397719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397528 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.397719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397689 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43be8571-61d3-47ea-aeb5-67d131720b19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.397835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397758 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.397835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.397818 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz485\" (UniqueName: \"kubernetes.io/projected/43be8571-61d3-47ea-aeb5-67d131720b19-kube-api-access-tz485\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.498952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.498907 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43be8571-61d3-47ea-aeb5-67d131720b19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499002 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499161 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz485\" (UniqueName: \"kubernetes.io/projected/43be8571-61d3-47ea-aeb5-67d131720b19-kube-api-access-tz485\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499229 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499290 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499421 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.499672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499647 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.500144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.499714 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.502338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.502309 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43be8571-61d3-47ea-aeb5-67d131720b19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.503019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.502998 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43be8571-61d3-47ea-aeb5-67d131720b19-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.525646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.525623 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz485\" (UniqueName: \"kubernetes.io/projected/43be8571-61d3-47ea-aeb5-67d131720b19-kube-api-access-tz485\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-sjb24\" (UID: \"43be8571-61d3-47ea-aeb5-67d131720b19\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:00.627506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:00.627437 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:01.252517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:01.164242 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24"] Nov 03 16:33:01.582063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:01.581982 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" event={"ID":"43be8571-61d3-47ea-aeb5-67d131720b19","Type":"ContainerStarted","Data":"e34c0b4d474f8097df0b58e0e62e933f6d6e3459ba04bb009ddb5007e111259a"} Nov 03 16:33:02.587801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:02.587764 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" event={"ID":"43be8571-61d3-47ea-aeb5-67d131720b19","Type":"ContainerStarted","Data":"c03118964a97274ba9b0623c93636515d205b84531347ea8896e2fef2b2224f9"} Nov 03 16:33:02.628005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:02.627971 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:02.635555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:02.635511 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" podStartSLOduration=2.347390141 podStartE2EDuration="2.6354972s" totalImagesPullingTime="288.107059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:02.634551185 +0000 UTC m=+1077.967764742" watchObservedRunningTime="2026-11-03 16:33:02.6354972 +0000 UTC m=+1077.968710768" Nov 03 16:33:06.119484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.118984 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-njjrb"] Nov 03 16:33:06.163573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.163517 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.166867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.166821 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-home\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.167166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.167129 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf9p2\" (UniqueName: \"kubernetes.io/projected/28008665-4e16-49dd-acc6-3e312861194d-kube-api-access-bf9p2\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.167323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.167305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.167485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.167468 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.167738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.167712 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28008665-4e16-49dd-acc6-3e312861194d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.167931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.167914 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.187055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.182090 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.260515 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-njjrb"] Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.271995 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28008665-4e16-49dd-acc6-3e312861194d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.272068 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.272124 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-home\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.272153 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf9p2\" (UniqueName: \"kubernetes.io/projected/28008665-4e16-49dd-acc6-3e312861194d-kube-api-access-bf9p2\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.272185 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.272217 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.289894 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-home\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.290168 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.303475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.290780 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.516466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.516411 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf9p2\" (UniqueName: \"kubernetes.io/projected/28008665-4e16-49dd-acc6-3e312861194d-kube-api-access-bf9p2\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.520463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.516896 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28008665-4e16-49dd-acc6-3e312861194d-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.521204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.521178 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28008665-4e16-49dd-acc6-3e312861194d-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-njjrb\" (UID: \"28008665-4e16-49dd-acc6-3e312861194d\") " pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:06.530617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:06.530583 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:07.125260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:07.125186 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:33:07.130785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:07.130764 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-njjrb"] Nov 03 16:33:07.133502 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:33:07.133472 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28008665_4e16_49dd_acc6_3e312861194d.slice/crio-6b053fefcdf95737d276326ab76a3d0b77ece0a99f3a97d2d7e40cc2336f3c0a WatchSource:0}: Error finding container 6b053fefcdf95737d276326ab76a3d0b77ece0a99f3a97d2d7e40cc2336f3c0a: Status 404 returned error can't find the container with id 6b053fefcdf95737d276326ab76a3d0b77ece0a99f3a97d2d7e40cc2336f3c0a Nov 03 16:33:07.138503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:07.138479 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:33:07.677604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:07.677570 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" event={"ID":"28008665-4e16-49dd-acc6-3e312861194d","Type":"ContainerStarted","Data":"6b053fefcdf95737d276326ab76a3d0b77ece0a99f3a97d2d7e40cc2336f3c0a"} Nov 03 16:33:07.689158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:07.689130 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" Nov 03 16:33:10.628409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:10.628370 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:10.646597 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:10.646564 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:10.716251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:10.716208 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" event={"ID":"28008665-4e16-49dd-acc6-3e312861194d","Type":"ContainerStarted","Data":"42a8c155c19726097b6f367d9660cc0f1775785ad10ea5b025ccc0fa2eb7aef7"} Nov 03 16:33:10.727505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:10.727478 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" Nov 03 16:33:10.767987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:10.767939 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" podStartSLOduration=1.361400583 podStartE2EDuration="4.767925279s" totalImagesPullingTime="3.406524696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:10.766834029 +0000 UTC m=+1086.100047598" watchObservedRunningTime="2026-11-03 16:33:10.767925279 +0000 UTC m=+1086.101138959" Nov 03 16:33:11.539479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:11.539425 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:12.706345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.706309 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l"] Nov 03 16:33:12.710950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.710925 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.740814 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.740779 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 16:33:12.756598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.756563 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l"] Nov 03 16:33:12.806192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806157 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.806192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806195 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.806397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806330 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.806397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806384 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.806492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806414 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r8n\" (UniqueName: \"kubernetes.io/projected/a7aa56f7-e028-49dc-b1df-c6437a0096d6-kube-api-access-p4r8n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.806533 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.806501 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.907892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.907853 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r8n\" (UniqueName: \"kubernetes.io/projected/a7aa56f7-e028-49dc-b1df-c6437a0096d6-kube-api-access-p4r8n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.907919 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908001 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908031 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908100 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908138 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908415 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908498 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.908591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.908573 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.910493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.910465 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7aa56f7-e028-49dc-b1df-c6437a0096d6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.910632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.910614 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7aa56f7-e028-49dc-b1df-c6437a0096d6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:12.935570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:12.935541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r8n\" (UniqueName: \"kubernetes.io/projected/a7aa56f7-e028-49dc-b1df-c6437a0096d6-kube-api-access-p4r8n\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l\" (UID: \"a7aa56f7-e028-49dc-b1df-c6437a0096d6\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:13.025251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:13.025182 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:13.342740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:13.342691 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l"] Nov 03 16:33:13.344118 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:33:13.344073 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7aa56f7_e028_49dc_b1df_c6437a0096d6.slice/crio-91fcf73665810ad88392172954c32e2bdfd3277d58f37e05fbacbd45d7446498 WatchSource:0}: Error finding container 91fcf73665810ad88392172954c32e2bdfd3277d58f37e05fbacbd45d7446498: Status 404 returned error can't find the container with id 91fcf73665810ad88392172954c32e2bdfd3277d58f37e05fbacbd45d7446498 Nov 03 16:33:13.740183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:13.740139 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" event={"ID":"a7aa56f7-e028-49dc-b1df-c6437a0096d6","Type":"ContainerStarted","Data":"e41865430c7f28110c77bdd969f82b73185e13985e058c816e12eefabf61a30b"} Nov 03 16:33:13.740708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:13.740197 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" event={"ID":"a7aa56f7-e028-49dc-b1df-c6437a0096d6","Type":"ContainerStarted","Data":"91fcf73665810ad88392172954c32e2bdfd3277d58f37e05fbacbd45d7446498"} Nov 03 16:33:13.788908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:13.788815 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podStartSLOduration=1.52715743 podStartE2EDuration="1.788793309s" totalImagesPullingTime="261.635879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:13.786155136 +0000 UTC m=+1089.119368707" watchObservedRunningTime="2026-11-03 16:33:13.788793309 +0000 UTC m=+1089.122006879" Nov 03 16:33:14.028527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:14.028481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:16.539127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:16.539047 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:16.551891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:16.551866 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:16.791908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:16.791817 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" Nov 03 16:33:22.249743 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:33:22.249709 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:33:23.025893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:23.025863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:23.038552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:23.038518 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:23.869022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:23.868994 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:33:32.136545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.136508 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4c8d89d8-khtcc"] Nov 03 16:33:32.142666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.142640 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.161435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.161402 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4c8d89d8-khtcc"] Nov 03 16:33:32.172702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.172675 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 16:33:32.172875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.172768 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-qvhsq\"" Nov 03 16:33:32.286123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.286085 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6gpd\" (UniqueName: \"kubernetes.io/projected/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-kube-api-access-b6gpd\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.286313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.286234 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-tls-cert\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.387695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.387620 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6gpd\" (UniqueName: \"kubernetes.io/projected/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-kube-api-access-b6gpd\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.387861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.387737 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-tls-cert\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.390179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.390152 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-tls-cert\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.411675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.411652 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6gpd\" (UniqueName: \"kubernetes.io/projected/7ac3d41f-22e1-4ef3-8389-4fc3e73d5196-kube-api-access-b6gpd\") pod \"authorino-6d4c8d89d8-khtcc\" (UID: \"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196\") " pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.455690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.455650 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" Nov 03 16:33:32.758062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.758032 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4c8d89d8-khtcc"] Nov 03 16:33:32.758513 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:33:32.758484 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ac3d41f_22e1_4ef3_8389_4fc3e73d5196.slice/crio-2d32513a24e91d59ae1ed79da8720ade2a1d49290a1cad59fa1f1a60352d563a WatchSource:0}: Error finding container 2d32513a24e91d59ae1ed79da8720ade2a1d49290a1cad59fa1f1a60352d563a: Status 404 returned error can't find the container with id 2d32513a24e91d59ae1ed79da8720ade2a1d49290a1cad59fa1f1a60352d563a Nov 03 16:33:32.903921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:32.903889 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" event={"ID":"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196","Type":"ContainerStarted","Data":"2d32513a24e91d59ae1ed79da8720ade2a1d49290a1cad59fa1f1a60352d563a"} Nov 03 16:33:33.911565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:33.911473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" event={"ID":"7ac3d41f-22e1-4ef3-8389-4fc3e73d5196","Type":"ContainerStarted","Data":"64ac020e66070fa66b5c5050aa3326eea641ec28d7cd80bcbe45e5675803347c"} Nov 03 16:33:33.953543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:33:33.953487 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d4c8d89d8-khtcc" podStartSLOduration=1.117873422 podStartE2EDuration="1.953470121s" totalImagesPullingTime="835.596699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:33.953210404 +0000 UTC m=+1109.286423974" watchObservedRunningTime="2026-11-03 16:33:33.953470121 +0000 UTC m=+1109.286683687" Nov 03 16:33:52.249668 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:33:52.249631 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:34:06.936413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:06.936379 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:06.940968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:06.940942 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:06.966040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:06.966014 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cacca478-gw-openshift-default-dockercfg-bk94f\"" Nov 03 16:34:06.980501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:06.980438 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:07.032254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032223 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032290 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032313 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjx4\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032334 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032356 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032494 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032550 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032619 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.032681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.032656 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133534 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133565 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjx4\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133591 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133616 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133644 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133696 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133730 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.133834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133772 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.133836 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.134119 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.134272 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.134311 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.134384 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.134508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.134407 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.136142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.136123 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.136347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.136330 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.157950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.157924 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.158165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.158143 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjx4\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\") pod \"e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.254922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.254876 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:07.565184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.565158 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:07.566415 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:07.566388 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e77f7c7_840e_48e7_a272_13a3cba74ab9.slice/crio-7aac21ab722a538736333459f2ce9bd0d9319a3be9273241684e2f3823997f0a WatchSource:0}: Error finding container 7aac21ab722a538736333459f2ce9bd0d9319a3be9273241684e2f3823997f0a: Status 404 returned error can't find the container with id 7aac21ab722a538736333459f2ce9bd0d9319a3be9273241684e2f3823997f0a Nov 03 16:34:07.568642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.568612 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:07.568712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.568675 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:07.568712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:07.568704 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:08.028957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.028924 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:08.032850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.032824 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.048968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.048944 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b5770a0e-openshift-default-dockercfg-8zv4n\"" Nov 03 16:34:08.064532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.064506 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:08.087774 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.087737 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerStarted","Data":"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda"} Nov 03 16:34:08.087901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.087801 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerStarted","Data":"7aac21ab722a538736333459f2ce9bd0d9319a3be9273241684e2f3823997f0a"} Nov 03 16:34:08.137343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.137295 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podStartSLOduration=2.137279968 podStartE2EDuration="2.137279968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:08.136571687 +0000 UTC m=+1143.469785256" watchObservedRunningTime="2026-11-03 16:34:08.137279968 +0000 UTC m=+1143.470493520" Nov 03 16:34:08.145792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.145760 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.145933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.145813 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.145933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.145908 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.145937 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.146104 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcgr\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.146180 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.146209 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.146270 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.146565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.146541 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.247898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.247865 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.247898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.247902 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.247921 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.247962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.247992 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248028 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248057 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248082 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248203 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcgr\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248405 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248479 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248688 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248800 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.248878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.248855 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.250147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.250125 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.250558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.250528 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.255347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.255322 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:08.261750 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.261672 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:08.292650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.292538 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcgr\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.306015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.305983 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\") pod \"e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.347155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.347120 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:08.659883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.659851 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:08.660549 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:08.660523 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec3b3d6_f6ad_4259_ad52_9b56686ffc73.slice/crio-47ec4dd4ee1e0de34b90b9c1ccd6a0f1f1c7f7bec42950987898ac0e2fc42a1f WatchSource:0}: Error finding container 47ec4dd4ee1e0de34b90b9c1ccd6a0f1f1c7f7bec42950987898ac0e2fc42a1f: Status 404 returned error can't find the container with id 47ec4dd4ee1e0de34b90b9c1ccd6a0f1f1c7f7bec42950987898ac0e2fc42a1f Nov 03 16:34:08.662784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.662754 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:08.662870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.662828 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:08.662870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:08.662857 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:09.095280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.095243 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerStarted","Data":"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b"} Nov 03 16:34:09.095674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.095296 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerStarted","Data":"47ec4dd4ee1e0de34b90b9c1ccd6a0f1f1c7f7bec42950987898ac0e2fc42a1f"} Nov 03 16:34:09.095674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.095474 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:09.096431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.096415 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:09.144099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.144048 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podStartSLOduration=2.1440328 podStartE2EDuration="2.1440328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:09.142972929 +0000 UTC m=+1144.476186499" watchObservedRunningTime="2026-11-03 16:34:09.1440328 +0000 UTC m=+1144.477246369" Nov 03 16:34:09.347303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.347223 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:09.352350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:09.352325 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:10.100714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:10.100674 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:10.101566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:10.101548 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:11.092255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.092219 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:11.096169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.096151 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.116838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.116816 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:11.128175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.128154 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cacca478\"" Nov 03 16:34:11.128297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.128185 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cacca478-dockercfg-k4hdg\"" Nov 03 16:34:11.163470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.162981 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:11.171892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.169225 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.182880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.182063 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqxl\" (UniqueName: \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.182880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.182808 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.188906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.188865 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:11.283947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.283901 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.284166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.283991 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqxl\" (UniqueName: \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.284166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.284037 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkdb\" (UniqueName: \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.284166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.284081 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.284813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.284786 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.310420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.310387 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqxl\" (UniqueName: \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\") pod \"payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.384909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.384817 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkdb\" (UniqueName: \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.384909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.384869 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.385579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.385560 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.410312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.410279 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:11.411332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.411298 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkdb\" (UniqueName: \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\") pod \"payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.493211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.493093 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:11.715494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.715355 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:11.727527 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:11.721184 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bde85e7_5b1f_4022_83ba_b4b4e28c8295.slice/crio-c0e4e45fe75e6b2bb4579038e4bfa979b43d40cddf6fc73740bb4eb1518b5907 WatchSource:0}: Error finding container c0e4e45fe75e6b2bb4579038e4bfa979b43d40cddf6fc73740bb4eb1518b5907: Status 404 returned error can't find the container with id c0e4e45fe75e6b2bb4579038e4bfa979b43d40cddf6fc73740bb4eb1518b5907 Nov 03 16:34:11.798689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:11.798665 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:11.799907 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:11.799877 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode799d8dc_3e72_47dd_8d24_474d4cfba836.slice/crio-3e96493518cd7e1b95c1ca00c421a7fc1cfc24c7440f101e1d521f5d8a13509c WatchSource:0}: Error finding container 3e96493518cd7e1b95c1ca00c421a7fc1cfc24c7440f101e1d521f5d8a13509c: Status 404 returned error can't find the container with id 3e96493518cd7e1b95c1ca00c421a7fc1cfc24c7440f101e1d521f5d8a13509c Nov 03 16:34:12.119866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.119825 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" event={"ID":"e799d8dc-3e72-47dd-8d24-474d4cfba836","Type":"ContainerStarted","Data":"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20"} Nov 03 16:34:12.120297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.119889 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" event={"ID":"e799d8dc-3e72-47dd-8d24-474d4cfba836","Type":"ContainerStarted","Data":"3e96493518cd7e1b95c1ca00c421a7fc1cfc24c7440f101e1d521f5d8a13509c"} Nov 03 16:34:12.120297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.120022 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:12.123582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.121739 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" event={"ID":"5bde85e7-5b1f-4022-83ba-b4b4e28c8295","Type":"ContainerStarted","Data":"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c"} Nov 03 16:34:12.123582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.121798 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" event={"ID":"5bde85e7-5b1f-4022-83ba-b4b4e28c8295","Type":"ContainerStarted","Data":"c0e4e45fe75e6b2bb4579038e4bfa979b43d40cddf6fc73740bb4eb1518b5907"} Nov 03 16:34:12.164244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.164185 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" podStartSLOduration=1.164167588 podStartE2EDuration="1.164167588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:12.163936441 +0000 UTC m=+1147.497150014" watchObservedRunningTime="2026-11-03 16:34:12.164167588 +0000 UTC m=+1147.497381159" Nov 03 16:34:12.212656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.212607 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" podStartSLOduration=1.212593939 podStartE2EDuration="1.212593939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:12.210751691 +0000 UTC m=+1147.543965259" watchObservedRunningTime="2026-11-03 16:34:12.212593939 +0000 UTC m=+1147.545807508" Nov 03 16:34:12.950648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.950603 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:12.956221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.956188 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:12.972330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.972302 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:12.983536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.983514 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f0b922f4-dockercfg-9sj2t\"" Nov 03 16:34:12.983536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:12.983514 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f0b922f4\"" Nov 03 16:34:13.005544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.005512 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.005664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.005593 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2kkd\" (UniqueName: \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.034038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.034006 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:13.037885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.037858 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.060292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.060256 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:13.106831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.106784 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.108058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.108032 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6mnc\" (UniqueName: \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.108303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.108287 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2kkd\" (UniqueName: \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.108523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.108506 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.108773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.107974 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.127789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.127694 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:13.137735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.137693 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2kkd\" (UniqueName: \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\") pod \"payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.209823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.209780 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.210051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.210025 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6mnc\" (UniqueName: \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.211019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.210992 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.241162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.241119 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6mnc\" (UniqueName: \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\") pod \"payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.270124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.270088 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:13.353058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.353011 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:13.571951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.571859 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:13.670247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:13.670198 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:14.134081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.134042 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" event={"ID":"94d2668f-999f-4513-901b-3879fe17189e","Type":"ContainerStarted","Data":"bdb813b9e94abb3055642d1e94fcb87eecf2cb003d6288eb2104745c56d4db6a"} Nov 03 16:34:14.134591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.134107 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" event={"ID":"94d2668f-999f-4513-901b-3879fe17189e","Type":"ContainerStarted","Data":"79acdad69a8e5171a424ef5566009af3b11c49b1e23f8b397d5b9c5894f5ab48"} Nov 03 16:34:14.134591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.134231 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:14.135722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.135699 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" event={"ID":"4a9515a0-1b33-4b43-ae16-519867a5c39d","Type":"ContainerStarted","Data":"443af20428fe6b41b22a4f486f41dfab25db1bc456da0660d808c5f296298f30"} Nov 03 16:34:14.135839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.135741 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" event={"ID":"4a9515a0-1b33-4b43-ae16-519867a5c39d","Type":"ContainerStarted","Data":"a31aae2139a96348d1f47e780f203aeef5057a4f93823491af98754677ebe27b"} Nov 03 16:34:14.195683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.195631 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" podStartSLOduration=2.195615367 podStartE2EDuration="2.195615367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:14.193614979 +0000 UTC m=+1149.526828547" watchObservedRunningTime="2026-11-03 16:34:14.195615367 +0000 UTC m=+1149.528828936" Nov 03 16:34:14.239473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.239391 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" podStartSLOduration=1.239377312 podStartE2EDuration="1.239377312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:14.238298449 +0000 UTC m=+1149.571512020" watchObservedRunningTime="2026-11-03 16:34:14.239377312 +0000 UTC m=+1149.572590880" Nov 03 16:34:14.790414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.790380 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:14.794972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.794947 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:14.819303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.819270 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b5770a0e\"" Nov 03 16:34:14.821512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.821482 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:14.838056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.838024 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b5770a0e-dockercfg-999dt\"" Nov 03 16:34:14.883064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.883024 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:14.887183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.887158 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:14.906509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.906479 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:14.932910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.932871 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:14.933099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:14.932916 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svpgn\" (UniqueName: \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.033940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.033901 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.033940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.033940 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svpgn\" (UniqueName: \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.034158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.033965 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np224\" (UniqueName: \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.034200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.034159 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.034864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.034842 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.068869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.068775 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svpgn\" (UniqueName: \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\") pod \"payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.109662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.109624 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:15.135194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.135156 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np224\" (UniqueName: \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.135575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.135302 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.136082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.136057 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.141117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.141091 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:15.172468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.171954 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np224\" (UniqueName: \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\") pod \"payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.200871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.200842 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:15.433118 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.433084 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:15.433249 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:15.433226 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b336bb3_be36_42e4_bff0_f6e1d52a9e25.slice/crio-653b1ca66a0bbffd61f95fadb493c855fe10150a7a7589eaa673ebc826cd6d1e WatchSource:0}: Error finding container 653b1ca66a0bbffd61f95fadb493c855fe10150a7a7589eaa673ebc826cd6d1e: Status 404 returned error can't find the container with id 653b1ca66a0bbffd61f95fadb493c855fe10150a7a7589eaa673ebc826cd6d1e Nov 03 16:34:15.536205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:15.536177 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:15.536309 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:15.536197 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fd9ef6_e85a_48d4_a8cd_8f175594d16c.slice/crio-c78468aaf982e760b72e24762b71667d2d7c0fd8c89742f01d53aa50f3356cb1 WatchSource:0}: Error finding container c78468aaf982e760b72e24762b71667d2d7c0fd8c89742f01d53aa50f3356cb1: Status 404 returned error can't find the container with id c78468aaf982e760b72e24762b71667d2d7c0fd8c89742f01d53aa50f3356cb1 Nov 03 16:34:16.175203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.175160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" event={"ID":"78fd9ef6-e85a-48d4-a8cd-8f175594d16c","Type":"ContainerStarted","Data":"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1"} Nov 03 16:34:16.175712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.175222 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" event={"ID":"78fd9ef6-e85a-48d4-a8cd-8f175594d16c","Type":"ContainerStarted","Data":"c78468aaf982e760b72e24762b71667d2d7c0fd8c89742f01d53aa50f3356cb1"} Nov 03 16:34:16.175712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.175326 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:16.176839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.176810 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" event={"ID":"3b336bb3-be36-42e4-bff0-f6e1d52a9e25","Type":"ContainerStarted","Data":"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d"} Nov 03 16:34:16.176945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.176863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:16.176945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.176884 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" event={"ID":"3b336bb3-be36-42e4-bff0-f6e1d52a9e25","Type":"ContainerStarted","Data":"653b1ca66a0bbffd61f95fadb493c855fe10150a7a7589eaa673ebc826cd6d1e"} Nov 03 16:34:16.219215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.219162 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" podStartSLOduration=2.219148494 podStartE2EDuration="2.219148494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:16.218111438 +0000 UTC m=+1151.551325007" watchObservedRunningTime="2026-11-03 16:34:16.219148494 +0000 UTC m=+1151.552362064" Nov 03 16:34:16.263806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:16.263730 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" podStartSLOduration=2.263711848 podStartE2EDuration="2.263711848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:16.262896167 +0000 UTC m=+1151.596109749" watchObservedRunningTime="2026-11-03 16:34:16.263711848 +0000 UTC m=+1151.596925418" Nov 03 16:34:22.251331 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:22.251281 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:34:23.131830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.131798 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:23.219341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.219308 6001 generic.go:356] "Generic (PLEG): container finished" podID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerID="9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda" exitCode=0 Nov 03 16:34:23.219561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.219383 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerDied","Data":"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda"} Nov 03 16:34:23.219876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.219858 6001 scope.go:122] "RemoveContainer" containerID="9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda" Nov 03 16:34:23.221225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.221200 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:23.221293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.221269 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:23.221334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:23.221308 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:24.095691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.095642 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:24.138581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.138556 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:24.227655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.227615 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerStarted","Data":"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06"} Nov 03 16:34:24.266468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.257019 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:24.266468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.257182 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:24.266468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.257227 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:24.902511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.902403 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf"] Nov 03 16:34:24.906818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.906795 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:24.918862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.918831 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf"] Nov 03 16:34:24.929819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.929793 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:34:24.930001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.929870 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:34:24.930001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.929968 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:34:24.930655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:24.930637 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:34:25.008020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.007982 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kgtk\" (UniqueName: \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.008020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.008016 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.008236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.008181 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.109803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.109766 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kgtk\" (UniqueName: \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.109803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.109803 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.110297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.110001 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.110431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.110407 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.112336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.112314 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.134173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.134144 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kgtk\" (UniqueName: \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\") pod \"maas-api-revoke-keys-e2e-migrate-cacca478-8htgf\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.171700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.171610 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:25.221723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.221692 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:25.288469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.266636 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:25.305961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.305918 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:25.630475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:25.629734 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf"] Nov 03 16:34:25.630924 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:25.630897 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7fdf0b_841c_4273_bde1_486fb325e047.slice/crio-f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404 WatchSource:0}: Error finding container f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404: Status 404 returned error can't find the container with id f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404 Nov 03 16:34:26.190468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:26.190072 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:26.247471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:26.243503 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" event={"ID":"6b7fdf0b-841c-4273-bde1-486fb325e047","Type":"ContainerStarted","Data":"f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404"} Nov 03 16:34:26.269473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:26.260833 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:26.269473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:26.260893 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:27.199221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:27.198213 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:27.200724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:27.200592 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:27.265713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:27.265211 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:27.265713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:27.265265 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:28.270632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:28.270592 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:28.271032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:28.270653 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:29.255486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:29.255438 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:29.255661 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:29.255508 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Nov 03 16:34:30.259994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.259959 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:30.260410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.260281 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:30.261137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.261115 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:30.324646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.324605 6001 generic.go:356] "Generic (PLEG): container finished" podID="6b7fdf0b-841c-4273-bde1-486fb325e047" containerID="bcfc02349315a22cb84cd3c77b3385ef55ef251093f3492427cb5b4bee9a82aa" exitCode=0 Nov 03 16:34:30.324815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.324687 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" event={"ID":"6b7fdf0b-841c-4273-bde1-486fb325e047","Type":"ContainerDied","Data":"bcfc02349315a22cb84cd3c77b3385ef55ef251093f3492427cb5b4bee9a82aa"} Nov 03 16:34:30.709641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.709579 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg"] Nov 03 16:34:30.714097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.714074 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.735128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.735097 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg"] Nov 03 16:34:30.804010 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.803978 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.804175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.804023 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.804175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.804138 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8qg\" (UniqueName: \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.905004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.904962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.905177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.905023 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.905177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.905070 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8qg\" (UniqueName: \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.905805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.905780 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.907553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.907528 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:30.949826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:30.949786 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8qg\" (UniqueName: \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\") pod \"maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:31.029537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.029494 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:31.328183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.328149 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg"] Nov 03 16:34:31.329805 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:31.329777 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda252e3f1_3fc5_4556_9d6f_62cbf2fa064f.slice/crio-b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8 WatchSource:0}: Error finding container b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8: Status 404 returned error can't find the container with id b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8 Nov 03 16:34:31.468628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.468601 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:31.514428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.514396 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\" (UniqueName: \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\") pod \"6b7fdf0b-841c-4273-bde1-486fb325e047\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " Nov 03 16:34:31.514649 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.514513 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\") pod \"6b7fdf0b-841c-4273-bde1-486fb325e047\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " Nov 03 16:34:31.514649 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.514559 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\") pod \"6b7fdf0b-841c-4273-bde1-486fb325e047\" (UID: \"6b7fdf0b-841c-4273-bde1-486fb325e047\") " Nov 03 16:34:31.515311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.515279 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca" pod "6b7fdf0b-841c-4273-bde1-486fb325e047" (UID: "6b7fdf0b-841c-4273-bde1-486fb325e047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:31.518162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.518132 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca" pod "6b7fdf0b-841c-4273-bde1-486fb325e047" (UID: "6b7fdf0b-841c-4273-bde1-486fb325e047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:31.518301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.518284 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk" pod "6b7fdf0b-841c-4273-bde1-486fb325e047" (UID: "6b7fdf0b-841c-4273-bde1-486fb325e047"). InnerVolumeSpecName "kube-api-access-9kgtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:31.616587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.616498 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kgtk\" (UniqueName: \"kubernetes.io/projected/6b7fdf0b-841c-4273-bde1-486fb325e047-kube-api-access-9kgtk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:31.616587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.616532 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:31.616587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.616542 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7fdf0b-841c-4273-bde1-486fb325e047-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:31.828093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.828063 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277"] Nov 03 16:34:31.828620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.828604 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6b7fdf0b-841c-4273-bde1-486fb325e047" containerName="revoke-keys" Nov 03 16:34:31.828670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.828622 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7fdf0b-841c-4273-bde1-486fb325e047" containerName="revoke-keys" Nov 03 16:34:31.828749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.828737 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7fdf0b-841c-4273-bde1-486fb325e047" containerName="revoke-keys" Nov 03 16:34:31.832199 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.832179 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:31.852592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.852562 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277"] Nov 03 16:34:31.919927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.919890 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gmsr\" (UniqueName: \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:31.920096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.920072 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:31.920188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:31.920116 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.021153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.021108 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gmsr\" (UniqueName: \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.021311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.021222 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.021311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.021254 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.021999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.021977 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.023785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.023758 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.045529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.045500 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gmsr\" (UniqueName: \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\") pod \"maas-api-revoke-keys-e2e-ait-b5770a0e-6m277\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.148816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.148785 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:32.339062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.338974 6001 generic.go:356] "Generic (PLEG): container finished" podID="a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" containerID="0183a530bdba3ba907177d5fd0d7f1fb9b241d54b17b4db8bf27d07e375c04c6" exitCode=0 Nov 03 16:34:32.339538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.339063 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" event={"ID":"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f","Type":"ContainerDied","Data":"0183a530bdba3ba907177d5fd0d7f1fb9b241d54b17b4db8bf27d07e375c04c6"} Nov 03 16:34:32.339538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.339130 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" event={"ID":"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f","Type":"ContainerStarted","Data":"b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8"} Nov 03 16:34:32.340432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.340413 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" Nov 03 16:34:32.340560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.340455 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf" event={"ID":"6b7fdf0b-841c-4273-bde1-486fb325e047","Type":"ContainerDied","Data":"f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404"} Nov 03 16:34:32.340560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.340493 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2aa04133aeb13d4e6e10780dad8cc1597fbf0fcee3617c2118e9a4f6d231404" Nov 03 16:34:32.342133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.342111 6001 generic.go:356] "Generic (PLEG): container finished" podID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerID="c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b" exitCode=0 Nov 03 16:34:32.342211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.342185 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerDied","Data":"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b"} Nov 03 16:34:32.342573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.342557 6001 scope.go:122] "RemoveContainer" containerID="c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b" Nov 03 16:34:32.343971 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.343947 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:32.344035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.344019 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:32.344077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.344065 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:32.458409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:32.457103 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277"] Nov 03 16:34:32.471253 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:32.471020 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b26be8_a27d_4191_842d_e177bd976873.slice/crio-d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747 WatchSource:0}: Error finding container d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747: Status 404 returned error can't find the container with id d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747 Nov 03 16:34:33.371852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.371810 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" event={"ID":"e3b26be8-a27d-4191-842d-e177bd976873","Type":"ContainerStarted","Data":"24cfc9a15917557aa68f0f00b979cec5bb4378463ada4913f8dfd790305325f9"} Nov 03 16:34:33.372342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.371878 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" event={"ID":"e3b26be8-a27d-4191-842d-e177bd976873","Type":"ContainerStarted","Data":"d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747"} Nov 03 16:34:33.374157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.374128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerStarted","Data":"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc"} Nov 03 16:34:33.421694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.421640 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" podStartSLOduration=2.421625324 podStartE2EDuration="2.421625324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:33.420761101 +0000 UTC m=+1168.753974676" watchObservedRunningTime="2026-11-03 16:34:33.421625324 +0000 UTC m=+1168.754838942" Nov 03 16:34:33.536932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.536906 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:33.642183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.642094 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\") pod \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " Nov 03 16:34:33.642183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.642171 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\") pod \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " Nov 03 16:34:33.642383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.642225 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\" (UniqueName: \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\") pod \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\" (UID: \"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f\") " Nov 03 16:34:33.642513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.642490 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca" pod "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" (UID: "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:33.669258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.669145 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca" pod "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" (UID: "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:33.669258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.669150 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg" pod "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" (UID: "a252e3f1-3fc5-4556-9d6f-62cbf2fa064f"). InnerVolumeSpecName "kube-api-access-fk8qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:33.762469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.744120 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:33.762469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.744159 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:33.762469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:33.744174 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8qg\" (UniqueName: \"kubernetes.io/projected/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f-kube-api-access-fk8qg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:34.362469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.352433 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:34.362469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.357684 6001 patch_prober.go:29] interesting pod/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:34.362469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.357746 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:34:34.380798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.380767 6001 generic.go:356] "Generic (PLEG): container finished" podID="e3b26be8-a27d-4191-842d-e177bd976873" containerID="24cfc9a15917557aa68f0f00b979cec5bb4378463ada4913f8dfd790305325f9" exitCode=0 Nov 03 16:34:34.381195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.380855 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" event={"ID":"e3b26be8-a27d-4191-842d-e177bd976873","Type":"ContainerDied","Data":"24cfc9a15917557aa68f0f00b979cec5bb4378463ada4913f8dfd790305325f9"} Nov 03 16:34:34.382157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.382131 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" event={"ID":"a252e3f1-3fc5-4556-9d6f-62cbf2fa064f","Type":"ContainerDied","Data":"b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8"} Nov 03 16:34:34.382283 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.382175 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38e05835a0c12bd36b4ca51edf03459024684245cc7660192e2f36d439b7ff8" Nov 03 16:34:34.382283 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:34.382143 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg" Nov 03 16:34:35.363476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.347851 6001 patch_prober.go:29] interesting pod/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:35.363476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.347910 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:34:35.385463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.385418 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:35.410672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.410617 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:35.624256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.624188 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:35.684362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.684328 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\" (UniqueName: \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\") pod \"e3b26be8-a27d-4191-842d-e177bd976873\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " Nov 03 16:34:35.684534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.684422 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\") pod \"e3b26be8-a27d-4191-842d-e177bd976873\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " Nov 03 16:34:35.684601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.684586 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\") pod \"e3b26be8-a27d-4191-842d-e177bd976873\" (UID: \"e3b26be8-a27d-4191-842d-e177bd976873\") " Nov 03 16:34:35.685610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.685580 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca" pod "e3b26be8-a27d-4191-842d-e177bd976873" (UID: "e3b26be8-a27d-4191-842d-e177bd976873"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:35.687111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.687087 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca" pod "e3b26be8-a27d-4191-842d-e177bd976873" (UID: "e3b26be8-a27d-4191-842d-e177bd976873"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:35.689514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.689493 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr" pod "e3b26be8-a27d-4191-842d-e177bd976873" (UID: "e3b26be8-a27d-4191-842d-e177bd976873"). InnerVolumeSpecName "kube-api-access-8gmsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:35.785718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.785686 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:35.785718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.785717 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gmsr\" (UniqueName: \"kubernetes.io/projected/e3b26be8-a27d-4191-842d-e177bd976873-kube-api-access-8gmsr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:35.785911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:35.785727 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b26be8-a27d-4191-842d-e177bd976873-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:36.366468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:36.356889 6001 patch_prober.go:29] interesting pod/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:36.366468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:36.356954 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:34:36.394303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:36.394267 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" event={"ID":"e3b26be8-a27d-4191-842d-e177bd976873","Type":"ContainerDied","Data":"d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747"} Nov 03 16:34:36.394771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:36.394351 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fb5ca10bd606ba5f440dacdd6607c18d8582995fb0c7f2c65e7154c67ba747" Nov 03 16:34:36.394771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:36.394286 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277" Nov 03 16:34:37.348858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:37.347882 6001 patch_prober.go:29] interesting pod/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:37.348858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:37.347955 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:34:38.349567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:38.349522 6001 patch_prober.go:29] interesting pod/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:38.350056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:38.349585 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Nov 03 16:34:39.351861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:39.351834 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:39.352269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:39.352123 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:39.352914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:39.352895 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:40.386138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.386096 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" podUID="e799d8dc-3e72-47dd-8d24-474d4cfba836" containerName="payload-processing" containerID="cri-o://383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20" gracePeriod=25 Nov 03 16:34:40.411354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.411327 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" podUID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" containerName="payload-pre-processing" containerID="cri-o://992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c" gracePeriod=25 Nov 03 16:34:40.686307 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.686277 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:40.708982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.708952 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:40.742035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.742004 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\" (UniqueName: \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\") pod \"e799d8dc-3e72-47dd-8d24-474d4cfba836\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " Nov 03 16:34:40.742235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.742132 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\") pod \"e799d8dc-3e72-47dd-8d24-474d4cfba836\" (UID: \"e799d8dc-3e72-47dd-8d24-474d4cfba836\") " Nov 03 16:34:40.742561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.742530 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume" pod "e799d8dc-3e72-47dd-8d24-474d4cfba836" (UID: "e799d8dc-3e72-47dd-8d24-474d4cfba836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:40.744999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.744979 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb" pod "e799d8dc-3e72-47dd-8d24-474d4cfba836" (UID: "e799d8dc-3e72-47dd-8d24-474d4cfba836"). InnerVolumeSpecName "kube-api-access-czkdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:40.842875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.842838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\" (UniqueName: \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\") pod \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " Nov 03 16:34:40.843068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.842935 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\") pod \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\" (UID: \"5bde85e7-5b1f-4022-83ba-b4b4e28c8295\") " Nov 03 16:34:40.843296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.843277 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkdb\" (UniqueName: \"kubernetes.io/projected/e799d8dc-3e72-47dd-8d24-474d4cfba836-kube-api-access-czkdb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:40.843361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.843306 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e799d8dc-3e72-47dd-8d24-474d4cfba836-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:40.843361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.843279 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume" pod "5bde85e7-5b1f-4022-83ba-b4b4e28c8295" (UID: "5bde85e7-5b1f-4022-83ba-b4b4e28c8295"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:40.845016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.844994 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl" pod "5bde85e7-5b1f-4022-83ba-b4b4e28c8295" (UID: "5bde85e7-5b1f-4022-83ba-b4b4e28c8295"). InnerVolumeSpecName "kube-api-access-9wqxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:40.945144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.945061 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqxl\" (UniqueName: \"kubernetes.io/projected/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-kube-api-access-9wqxl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:40.945144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:40.945088 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bde85e7-5b1f-4022-83ba-b4b4e28c8295-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:41.298849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.298814 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:41.330918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.330890 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:41.427859 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.427821 6001 generic.go:356] "Generic (PLEG): container finished" podID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" containerID="992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c" exitCode=0 Nov 03 16:34:41.428365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.427995 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" event={"ID":"5bde85e7-5b1f-4022-83ba-b4b4e28c8295","Type":"ContainerDied","Data":"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c"} Nov 03 16:34:41.428365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.428045 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" event={"ID":"5bde85e7-5b1f-4022-83ba-b4b4e28c8295","Type":"ContainerDied","Data":"c0e4e45fe75e6b2bb4579038e4bfa979b43d40cddf6fc73740bb4eb1518b5907"} Nov 03 16:34:41.428365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.428074 6001 scope.go:122] "RemoveContainer" containerID="992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c" Nov 03 16:34:41.428365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.428269 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr" Nov 03 16:34:41.449308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.449265 6001 generic.go:356] "Generic (PLEG): container finished" podID="e799d8dc-3e72-47dd-8d24-474d4cfba836" containerID="383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20" exitCode=0 Nov 03 16:34:41.449489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.449333 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" event={"ID":"e799d8dc-3e72-47dd-8d24-474d4cfba836","Type":"ContainerDied","Data":"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20"} Nov 03 16:34:41.449489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.449385 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" Nov 03 16:34:41.449489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.449473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl" event={"ID":"e799d8dc-3e72-47dd-8d24-474d4cfba836","Type":"ContainerDied","Data":"3e96493518cd7e1b95c1ca00c421a7fc1cfc24c7440f101e1d521f5d8a13509c"} Nov 03 16:34:41.458056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.458039 6001 scope.go:122] "RemoveContainer" containerID="992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c" Nov 03 16:34:41.458306 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:41.458285 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c\": container with ID starting with 992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c not found: ID does not exist" containerID="992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c" Nov 03 16:34:41.458386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.458319 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c"} err="failed to get container status \"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c\": rpc error: code = NotFound desc = could not find container \"992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c\": container with ID starting with 992feca4e613326f93dabdcdad59867199816fc45a02fd2df1d51cd3b3d2a53c not found: ID does not exist" Nov 03 16:34:41.458386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.458343 6001 scope.go:122] "RemoveContainer" containerID="383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20" Nov 03 16:34:41.467178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.467155 6001 scope.go:122] "RemoveContainer" containerID="383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20" Nov 03 16:34:41.467520 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:41.467498 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20\": container with ID starting with 383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20 not found: ID does not exist" containerID="383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20" Nov 03 16:34:41.467575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.467529 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20"} err="failed to get container status \"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20\": rpc error: code = NotFound desc = could not find container \"383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20\": container with ID starting with 383b68c9814a6af88d1cecf45834fb5f363bffdc3102d5fc883844788da54f20 not found: ID does not exist" Nov 03 16:34:41.488194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.488161 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:41.492701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.492663 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-cacca478-5fcbc8ddbc-7p6dr"] Nov 03 16:34:41.527559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.527529 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:41.531895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:41.531870 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cacca478-dd5687445-pzvnl"] Nov 03 16:34:42.798644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:42.798610 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" path="/var/lib/kubelet/pods/5bde85e7-5b1f-4022-83ba-b4b4e28c8295/volumes" Nov 03 16:34:42.799198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:42.799181 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e799d8dc-3e72-47dd-8d24-474d4cfba836" path="/var/lib/kubelet/pods/e799d8dc-3e72-47dd-8d24-474d4cfba836/volumes" Nov 03 16:34:42.898145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:42.898114 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:42.933968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:42.933941 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:43.099967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.099887 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf"] Nov 03 16:34:43.102883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.102852 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cacca478-8htgf"] Nov 03 16:34:43.229218 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.229180 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:43.229439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.229413 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" containerID="cri-o://a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06" gracePeriod=30 Nov 03 16:34:43.455558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.455471 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg"] Nov 03 16:34:43.457883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.457858 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b922f4-gsnsg"] Nov 03 16:34:43.578285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.578247 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh"] Nov 03 16:34:43.578949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.578923 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" containerName="payload-pre-processing" Nov 03 16:34:43.578949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.578950 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" containerName="payload-pre-processing" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.578998 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e799d8dc-3e72-47dd-8d24-474d4cfba836" containerName="payload-processing" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579008 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e799d8dc-3e72-47dd-8d24-474d4cfba836" containerName="payload-processing" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579029 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e3b26be8-a27d-4191-842d-e177bd976873" containerName="revoke-keys" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579035 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b26be8-a27d-4191-842d-e177bd976873" containerName="revoke-keys" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579043 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" containerName="revoke-keys" Nov 03 16:34:43.579149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579049 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" containerName="revoke-keys" Nov 03 16:34:43.579419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579181 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b26be8-a27d-4191-842d-e177bd976873" containerName="revoke-keys" Nov 03 16:34:43.579419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579196 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" containerName="revoke-keys" Nov 03 16:34:43.579419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579210 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bde85e7-5b1f-4022-83ba-b4b4e28c8295" containerName="payload-pre-processing" Nov 03 16:34:43.579419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.579225 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e799d8dc-3e72-47dd-8d24-474d4cfba836" containerName="payload-processing" Nov 03 16:34:43.584223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.584204 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.599657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.599633 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:34:43.599837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.599816 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:34:43.599837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.599831 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:34:43.599944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.599816 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:34:43.609265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.609241 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh"] Nov 03 16:34:43.674003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.673973 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwvb\" (UniqueName: \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.674176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.674019 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.674176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.674130 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.775221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.775184 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwvb\" (UniqueName: \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.775400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.775234 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.775400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.775274 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.775879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.775852 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.777764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.777736 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.799204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.799179 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwvb\" (UniqueName: \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\") pod \"maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:43.897096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:43.897064 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:44.225284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:44.223766 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh"] Nov 03 16:34:44.225939 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:44.225700 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e89601a_ee5a_4b69_9c86_b94f014805a2.slice/crio-9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f WatchSource:0}: Error finding container 9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f: Status 404 returned error can't find the container with id 9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f Nov 03 16:34:44.472304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:44.472273 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" event={"ID":"5e89601a-ee5a-4b69-9c86-b94f014805a2","Type":"ContainerStarted","Data":"9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f"} Nov 03 16:34:44.788777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:44.788705 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7fdf0b-841c-4273-bde1-486fb325e047" path="/var/lib/kubelet/pods/6b7fdf0b-841c-4273-bde1-486fb325e047/volumes" Nov 03 16:34:44.789121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:44.789107 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a252e3f1-3fc5-4556-9d6f-62cbf2fa064f" path="/var/lib/kubelet/pods/a252e3f1-3fc5-4556-9d6f-62cbf2fa064f/volumes" Nov 03 16:34:45.262198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:45.262173 6001 patch_prober.go:29] interesting pod/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:34:45.262559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:45.262225 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:34:45.485861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:45.485820 6001 generic.go:356] "Generic (PLEG): container finished" podID="5e89601a-ee5a-4b69-9c86-b94f014805a2" containerID="86b805e4800ab9d594fcd3e04022927a6793869e65c9348d2fcccdc5a687c05c" exitCode=0 Nov 03 16:34:45.486036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:45.485902 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" event={"ID":"5e89601a-ee5a-4b69-9c86-b94f014805a2","Type":"ContainerDied","Data":"86b805e4800ab9d594fcd3e04022927a6793869e65c9348d2fcccdc5a687c05c"} Nov 03 16:34:46.300245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.300140 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" podUID="4a9515a0-1b33-4b43-ae16-519867a5c39d" containerName="payload-processing" containerID="cri-o://443af20428fe6b41b22a4f486f41dfab25db1bc456da0660d808c5f296298f30" gracePeriod=25 Nov 03 16:34:46.331201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.331113 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" podUID="94d2668f-999f-4513-901b-3879fe17189e" containerName="payload-pre-processing" containerID="cri-o://bdb813b9e94abb3055642d1e94fcb87eecf2cb003d6288eb2104745c56d4db6a" gracePeriod=25 Nov 03 16:34:46.493921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.493892 6001 generic.go:356] "Generic (PLEG): container finished" podID="94d2668f-999f-4513-901b-3879fe17189e" containerID="bdb813b9e94abb3055642d1e94fcb87eecf2cb003d6288eb2104745c56d4db6a" exitCode=0 Nov 03 16:34:46.494081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.493963 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" event={"ID":"94d2668f-999f-4513-901b-3879fe17189e","Type":"ContainerDied","Data":"bdb813b9e94abb3055642d1e94fcb87eecf2cb003d6288eb2104745c56d4db6a"} Nov 03 16:34:46.502671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.498724 6001 generic.go:356] "Generic (PLEG): container finished" podID="4a9515a0-1b33-4b43-ae16-519867a5c39d" containerID="443af20428fe6b41b22a4f486f41dfab25db1bc456da0660d808c5f296298f30" exitCode=0 Nov 03 16:34:46.502671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.499026 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" event={"ID":"4a9515a0-1b33-4b43-ae16-519867a5c39d","Type":"ContainerDied","Data":"443af20428fe6b41b22a4f486f41dfab25db1bc456da0660d808c5f296298f30"} Nov 03 16:34:46.636953 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.635779 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:46.683557 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.683520 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:46.710369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.710345 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:46.726478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.726434 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\" (UniqueName: \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\") pod \"4a9515a0-1b33-4b43-ae16-519867a5c39d\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " Nov 03 16:34:46.726641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.726507 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\") pod \"4a9515a0-1b33-4b43-ae16-519867a5c39d\" (UID: \"4a9515a0-1b33-4b43-ae16-519867a5c39d\") " Nov 03 16:34:46.726906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.726881 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume" pod "4a9515a0-1b33-4b43-ae16-519867a5c39d" (UID: "4a9515a0-1b33-4b43-ae16-519867a5c39d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:46.730075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.730045 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc" pod "4a9515a0-1b33-4b43-ae16-519867a5c39d" (UID: "4a9515a0-1b33-4b43-ae16-519867a5c39d"). InnerVolumeSpecName "kube-api-access-c6mnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:46.827786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.827707 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\" (UniqueName: \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\") pod \"94d2668f-999f-4513-901b-3879fe17189e\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " Nov 03 16:34:46.827786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.827747 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\") pod \"94d2668f-999f-4513-901b-3879fe17189e\" (UID: \"94d2668f-999f-4513-901b-3879fe17189e\") " Nov 03 16:34:46.827968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.827802 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\") pod \"5e89601a-ee5a-4b69-9c86-b94f014805a2\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " Nov 03 16:34:46.827968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.827875 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\" (UniqueName: \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\") pod \"5e89601a-ee5a-4b69-9c86-b94f014805a2\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " Nov 03 16:34:46.827968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.827894 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\") pod \"5e89601a-ee5a-4b69-9c86-b94f014805a2\" (UID: \"5e89601a-ee5a-4b69-9c86-b94f014805a2\") " Nov 03 16:34:46.828205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.828175 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume" pod "94d2668f-999f-4513-901b-3879fe17189e" (UID: "94d2668f-999f-4513-901b-3879fe17189e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:46.828205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.828192 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6mnc\" (UniqueName: \"kubernetes.io/projected/4a9515a0-1b33-4b43-ae16-519867a5c39d-kube-api-access-c6mnc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.828205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.828208 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9515a0-1b33-4b43-ae16-519867a5c39d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.828425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.828257 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca" pod "5e89601a-ee5a-4b69-9c86-b94f014805a2" (UID: "5e89601a-ee5a-4b69-9c86-b94f014805a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:46.830055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.830027 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca" pod "5e89601a-ee5a-4b69-9c86-b94f014805a2" (UID: "5e89601a-ee5a-4b69-9c86-b94f014805a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:46.830155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.830065 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb" pod "5e89601a-ee5a-4b69-9c86-b94f014805a2" (UID: "5e89601a-ee5a-4b69-9c86-b94f014805a2"). InnerVolumeSpecName "kube-api-access-grwvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:46.830155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.830103 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd" pod "94d2668f-999f-4513-901b-3879fe17189e" (UID: "94d2668f-999f-4513-901b-3879fe17189e"). InnerVolumeSpecName "kube-api-access-s2kkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:46.929309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.929274 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grwvb\" (UniqueName: \"kubernetes.io/projected/5e89601a-ee5a-4b69-9c86-b94f014805a2-kube-api-access-grwvb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.929309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.929306 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.929563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.929319 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2kkd\" (UniqueName: \"kubernetes.io/projected/94d2668f-999f-4513-901b-3879fe17189e-kube-api-access-s2kkd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.929563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.929331 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d2668f-999f-4513-901b-3879fe17189e-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:46.929563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:46.929343 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e89601a-ee5a-4b69-9c86-b94f014805a2-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:47.506846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.506813 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" Nov 03 16:34:47.507304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.506846 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz" event={"ID":"4a9515a0-1b33-4b43-ae16-519867a5c39d","Type":"ContainerDied","Data":"a31aae2139a96348d1f47e780f203aeef5057a4f93823491af98754677ebe27b"} Nov 03 16:34:47.507304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.506916 6001 scope.go:122] "RemoveContainer" containerID="443af20428fe6b41b22a4f486f41dfab25db1bc456da0660d808c5f296298f30" Nov 03 16:34:47.510651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.508528 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" Nov 03 16:34:47.510651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.508532 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh" event={"ID":"5e89601a-ee5a-4b69-9c86-b94f014805a2","Type":"ContainerDied","Data":"9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f"} Nov 03 16:34:47.510651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.508647 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9484cb699b4ec982c84b01c1c3c515538dacb66d1c56b77a477cbe3cb882d51f" Nov 03 16:34:47.510651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.510333 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" event={"ID":"94d2668f-999f-4513-901b-3879fe17189e","Type":"ContainerDied","Data":"79acdad69a8e5171a424ef5566009af3b11c49b1e23f8b397d5b9c5894f5ab48"} Nov 03 16:34:47.510651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.510418 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv" Nov 03 16:34:47.519596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.519515 6001 scope.go:122] "RemoveContainer" containerID="bdb813b9e94abb3055642d1e94fcb87eecf2cb003d6288eb2104745c56d4db6a" Nov 03 16:34:47.575918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.575824 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:47.588588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.588559 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b922f4-55c9b9c7fc-cpplz"] Nov 03 16:34:47.614743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.614712 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:47.618987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.618962 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b922f4-78f57fcd47-fqmjv"] Nov 03 16:34:47.899195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.899101 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" podUID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" containerName="payload-processing" containerID="cri-o://ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1" gracePeriod=25 Nov 03 16:34:47.934725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:47.934693 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" podUID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" containerName="payload-pre-processing" containerID="cri-o://36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d" gracePeriod=25 Nov 03 16:34:48.190562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.190527 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:48.217478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.217435 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:48.236762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.236727 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277"] Nov 03 16:34:48.250247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.250215 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b5770a0e-6m277"] Nov 03 16:34:48.346620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.346590 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\") pod \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " Nov 03 16:34:48.346767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.346632 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\" (UniqueName: \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\") pod \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\" (UID: \"78fd9ef6-e85a-48d4-a8cd-8f175594d16c\") " Nov 03 16:34:48.346767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.346740 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\") pod \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " Nov 03 16:34:48.346767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.346764 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\" (UniqueName: \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\") pod \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\" (UID: \"3b336bb3-be36-42e4-bff0-f6e1d52a9e25\") " Nov 03 16:34:48.347114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.347087 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume" pod "78fd9ef6-e85a-48d4-a8cd-8f175594d16c" (UID: "78fd9ef6-e85a-48d4-a8cd-8f175594d16c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:48.347219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.347143 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume" pod "3b336bb3-be36-42e4-bff0-f6e1d52a9e25" (UID: "3b336bb3-be36-42e4-bff0-f6e1d52a9e25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:48.348702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.348680 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224" pod "78fd9ef6-e85a-48d4-a8cd-8f175594d16c" (UID: "78fd9ef6-e85a-48d4-a8cd-8f175594d16c"). InnerVolumeSpecName "kube-api-access-np224". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:48.348808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.348790 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn" pod "3b336bb3-be36-42e4-bff0-f6e1d52a9e25" (UID: "3b336bb3-be36-42e4-bff0-f6e1d52a9e25"). InnerVolumeSpecName "kube-api-access-svpgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:48.448318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.448280 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.448318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.448320 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svpgn\" (UniqueName: \"kubernetes.io/projected/3b336bb3-be36-42e4-bff0-f6e1d52a9e25-kube-api-access-svpgn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.448548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.448331 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.448548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.448340 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np224\" (UniqueName: \"kubernetes.io/projected/78fd9ef6-e85a-48d4-a8cd-8f175594d16c-kube-api-access-np224\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.499835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.499808 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:48.516762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.516725 6001 generic.go:356] "Generic (PLEG): container finished" podID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" containerID="ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1" exitCode=0 Nov 03 16:34:48.517202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.516770 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" Nov 03 16:34:48.517202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.516803 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" event={"ID":"78fd9ef6-e85a-48d4-a8cd-8f175594d16c","Type":"ContainerDied","Data":"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1"} Nov 03 16:34:48.517202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.516856 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf" event={"ID":"78fd9ef6-e85a-48d4-a8cd-8f175594d16c","Type":"ContainerDied","Data":"c78468aaf982e760b72e24762b71667d2d7c0fd8c89742f01d53aa50f3356cb1"} Nov 03 16:34:48.517202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.516881 6001 scope.go:122] "RemoveContainer" containerID="ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1" Nov 03 16:34:48.518677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.518652 6001 generic.go:356] "Generic (PLEG): container finished" podID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerID="a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06" exitCode=0 Nov 03 16:34:48.518788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.518707 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" Nov 03 16:34:48.518788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.518715 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerDied","Data":"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06"} Nov 03 16:34:48.518910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.518784 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs" event={"ID":"0e77f7c7-840e-48e7-a272-13a3cba74ab9","Type":"ContainerDied","Data":"7aac21ab722a538736333459f2ce9bd0d9319a3be9273241684e2f3823997f0a"} Nov 03 16:34:48.520306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.520285 6001 generic.go:356] "Generic (PLEG): container finished" podID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" containerID="36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d" exitCode=0 Nov 03 16:34:48.520434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.520339 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" Nov 03 16:34:48.520434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.520361 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" event={"ID":"3b336bb3-be36-42e4-bff0-f6e1d52a9e25","Type":"ContainerDied","Data":"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d"} Nov 03 16:34:48.520434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.520407 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz" event={"ID":"3b336bb3-be36-42e4-bff0-f6e1d52a9e25","Type":"ContainerDied","Data":"653b1ca66a0bbffd61f95fadb493c855fe10150a7a7589eaa673ebc826cd6d1e"} Nov 03 16:34:48.527222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.527196 6001 scope.go:122] "RemoveContainer" containerID="ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1" Nov 03 16:34:48.527566 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:48.527543 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1\": container with ID starting with ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1 not found: ID does not exist" containerID="ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1" Nov 03 16:34:48.527632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.527574 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1"} err="failed to get container status \"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1\": rpc error: code = NotFound desc = could not find container \"ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1\": container with ID starting with ccf93868115f73025851a353595661af0fc2d38464d786cfd610bb2fb8f0c9e1 not found: ID does not exist" Nov 03 16:34:48.527632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.527594 6001 scope.go:122] "RemoveContainer" containerID="a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06" Nov 03 16:34:48.536972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.536954 6001 scope.go:122] "RemoveContainer" containerID="9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda" Nov 03 16:34:48.545756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.545735 6001 scope.go:122] "RemoveContainer" containerID="a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06" Nov 03 16:34:48.546025 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:48.546004 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06\": container with ID starting with a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06 not found: ID does not exist" containerID="a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06" Nov 03 16:34:48.546085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.546033 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06"} err="failed to get container status \"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06\": rpc error: code = NotFound desc = could not find container \"a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06\": container with ID starting with a3a5e56db3679ede2a1fb084cb536b58d3676be38319f96a49efda338bd40e06 not found: ID does not exist" Nov 03 16:34:48.546085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.546052 6001 scope.go:122] "RemoveContainer" containerID="9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda" Nov 03 16:34:48.546284 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:48.546269 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda\": container with ID starting with 9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda not found: ID does not exist" containerID="9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda" Nov 03 16:34:48.546326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.546289 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda"} err="failed to get container status \"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda\": rpc error: code = NotFound desc = could not find container \"9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda\": container with ID starting with 9fb39e9aa81abe019ad13ac7e3936e9e0a357ef8733f5cea5e6a7cf1ff4cacda not found: ID does not exist" Nov 03 16:34:48.546326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.546302 6001 scope.go:122] "RemoveContainer" containerID="36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d" Nov 03 16:34:48.558504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.556417 6001 scope.go:122] "RemoveContainer" containerID="36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d" Nov 03 16:34:48.558504 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:48.556799 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d\": container with ID starting with 36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d not found: ID does not exist" containerID="36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d" Nov 03 16:34:48.558504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.556824 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d"} err="failed to get container status \"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d\": rpc error: code = NotFound desc = could not find container \"36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d\": container with ID starting with 36c692552c54c72a2567cff048ee26b4c63ee36752268f9d8f3a1bb58e60ee5d not found: ID does not exist" Nov 03 16:34:48.603267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.603231 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:48.607686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.607646 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b5770a0e-7868dd4697-sc6vf"] Nov 03 16:34:48.642201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.642170 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:48.646706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.646678 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b5770a0e-589ccb59bc-4lxfz"] Nov 03 16:34:48.650287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650262 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650330 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650351 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650367 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650398 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650418 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650529 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650637 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:48.650701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650653 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650680 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\") pod \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\" (UID: \"0e77f7c7-840e-48e7-a272-13a3cba74ab9\") " Nov 03 16:34:48.650701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650682 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:48.650861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650753 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:48.650932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.650910 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:48.651051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.651029 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:48.651110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.651069 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.651110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.651089 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.651110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.651103 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.651247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.651115 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.652888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.652861 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:34:48.652995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.652925 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:48.652995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.652944 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "kube-api-access-5vjx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:48.652995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.652968 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token" pod "0e77f7c7-840e-48e7-a272-13a3cba74ab9" (UID: "0e77f7c7-840e-48e7-a272-13a3cba74ab9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:48.753430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.753388 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.753430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.753428 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.753765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.753483 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.753765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.753501 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.753765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.753515 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjx4\" (UniqueName: \"kubernetes.io/projected/0e77f7c7-840e-48e7-a272-13a3cba74ab9-kube-api-access-5vjx4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:48.787795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.787756 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" path="/var/lib/kubelet/pods/3b336bb3-be36-42e4-bff0-f6e1d52a9e25/volumes" Nov 03 16:34:48.788227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.788206 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9515a0-1b33-4b43-ae16-519867a5c39d" path="/var/lib/kubelet/pods/4a9515a0-1b33-4b43-ae16-519867a5c39d/volumes" Nov 03 16:34:48.788654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.788636 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" path="/var/lib/kubelet/pods/78fd9ef6-e85a-48d4-a8cd-8f175594d16c/volumes" Nov 03 16:34:48.789047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.789029 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d2668f-999f-4513-901b-3879fe17189e" path="/var/lib/kubelet/pods/94d2668f-999f-4513-901b-3879fe17189e/volumes" Nov 03 16:34:48.789802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.789781 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b26be8-a27d-4191-842d-e177bd976873" path="/var/lib/kubelet/pods/e3b26be8-a27d-4191-842d-e177bd976873/volumes" Nov 03 16:34:48.872759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.872726 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:48.877874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.877829 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cacca478-gw-openshift-default-d8c875ccc-c95hs"] Nov 03 16:34:48.895939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.895906 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:48.896210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:48.896173 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" containerID="cri-o://25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc" gracePeriod=30 Nov 03 16:34:50.210110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210071 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 16:34:50.210641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210620 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210644 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210657 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" containerName="payload-pre-processing" Nov 03 16:34:50.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210662 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" containerName="payload-pre-processing" Nov 03 16:34:50.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210699 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210706 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210723 6001 cpu_manager.go:414] "containerMap: removing container" podUID="94d2668f-999f-4513-901b-3879fe17189e" containerName="payload-pre-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210731 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2668f-999f-4513-901b-3879fe17189e" containerName="payload-pre-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210752 6001 cpu_manager.go:414] "containerMap: removing container" podUID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" containerName="payload-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210758 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" containerName="payload-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210765 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5e89601a-ee5a-4b69-9c86-b94f014805a2" containerName="revoke-keys" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210771 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e89601a-ee5a-4b69-9c86-b94f014805a2" containerName="revoke-keys" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210789 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4a9515a0-1b33-4b43-ae16-519867a5c39d" containerName="payload-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210795 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9515a0-1b33-4b43-ae16-519867a5c39d" containerName="payload-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210931 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210944 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b336bb3-be36-42e4-bff0-f6e1d52a9e25" containerName="payload-pre-processing" Nov 03 16:34:50.210947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210954 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" containerName="istio-proxy" Nov 03 16:34:50.211275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210961 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d2668f-999f-4513-901b-3879fe17189e" containerName="payload-pre-processing" Nov 03 16:34:50.211275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210967 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fd9ef6-e85a-48d4-a8cd-8f175594d16c" containerName="payload-processing" Nov 03 16:34:50.211275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210973 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e89601a-ee5a-4b69-9c86-b94f014805a2" containerName="revoke-keys" Nov 03 16:34:50.211275 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.210980 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9515a0-1b33-4b43-ae16-519867a5c39d" containerName="payload-processing" Nov 03 16:34:50.216272 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.216244 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.234052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.234022 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-71d7b4-openshift-default-dockercfg-5vhxc\"" Nov 03 16:34:50.252068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.252032 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 16:34:50.369778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.369736 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.369778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.369783 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82jw\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.369995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.369890 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.369995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.369924 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.370058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.370001 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.370093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.370083 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.370124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.370117 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.370163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.370148 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.370220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.370206 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.471654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471563 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82jw\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.471654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471624 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.471654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471652 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.471929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471749 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.471929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.471954 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472007 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472037 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472051 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472132 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472283 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472345 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472472 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.472606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.472582 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.474259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.474235 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.474582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.474562 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.495905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.495871 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.496084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.496059 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82jw\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.531847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.531810 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:50.787989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.787957 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e77f7c7-840e-48e7-a272-13a3cba74ab9" path="/var/lib/kubelet/pods/0e77f7c7-840e-48e7-a272-13a3cba74ab9/volumes" Nov 03 16:34:50.817197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.817162 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 16:34:50.822048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.822024 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:50.842504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.842465 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 16:34:50.843189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.843169 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7519b7\"" Nov 03 16:34:50.843312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.843231 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7519b7-dockercfg-kk94s\"" Nov 03 16:34:50.865305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.865275 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 16:34:50.866467 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:50.866422 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb827585b_ffe1_4285_9a2f_7512655ebfdb.slice/crio-486ce76ff31c7a5ee6b8156c5020c8070561fe65fa066188c94daec8c636a9e9 WatchSource:0}: Error finding container 486ce76ff31c7a5ee6b8156c5020c8070561fe65fa066188c94daec8c636a9e9: Status 404 returned error can't find the container with id 486ce76ff31c7a5ee6b8156c5020c8070561fe65fa066188c94daec8c636a9e9 Nov 03 16:34:50.869056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.869022 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:50.869118 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.869099 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:50.869156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.869141 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:50.927114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.927074 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 16:34:50.933264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.933224 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:50.954648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.954602 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 16:34:50.979085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.979036 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crfhq\" (UniqueName: \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:50.979256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:50.979157 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.080582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.080484 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmqx\" (UniqueName: \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.080716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.080652 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.080763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.080746 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crfhq\" (UniqueName: \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.080829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.080815 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.081503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.081482 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.106624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.106588 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crfhq\" (UniqueName: \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\") pod \"payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.136315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.136275 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.182200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.182159 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmqx\" (UniqueName: \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.182422 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.182291 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.183245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.183218 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.206827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.206791 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmqx\" (UniqueName: \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\") pod \"payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.249671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.249628 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:51.436103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.436075 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 16:34:51.436940 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:51.436909 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a8048_eac8_4b9f_b676_f9bbd213bf0a.slice/crio-068629b7a68afcbbea56d315c73a5bec3115fca8de83f2efa432626dcc2521b4 WatchSource:0}: Error finding container 068629b7a68afcbbea56d315c73a5bec3115fca8de83f2efa432626dcc2521b4: Status 404 returned error can't find the container with id 068629b7a68afcbbea56d315c73a5bec3115fca8de83f2efa432626dcc2521b4 Nov 03 16:34:51.543942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.543913 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" event={"ID":"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a","Type":"ContainerStarted","Data":"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc"} Nov 03 16:34:51.544084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.543962 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" event={"ID":"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a","Type":"ContainerStarted","Data":"068629b7a68afcbbea56d315c73a5bec3115fca8de83f2efa432626dcc2521b4"} Nov 03 16:34:51.544130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.544110 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:34:51.545890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.545829 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerStarted","Data":"9050790ed792e206c05e678d31600a34e7312d57810bc9906548682f6681f029"} Nov 03 16:34:51.546068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.545917 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerStarted","Data":"486ce76ff31c7a5ee6b8156c5020c8070561fe65fa066188c94daec8c636a9e9"} Nov 03 16:34:51.549972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.549949 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 16:34:51.550751 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:51.550710 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a578f35_4d6b_4672_b34e_cc18a37c7cea.slice/crio-eefdf6f69275e32b6497d6c9eeac1b4d6976fb0d8b1cc3b2d81f1cc13e440c6d WatchSource:0}: Error finding container eefdf6f69275e32b6497d6c9eeac1b4d6976fb0d8b1cc3b2d81f1cc13e440c6d: Status 404 returned error can't find the container with id eefdf6f69275e32b6497d6c9eeac1b4d6976fb0d8b1cc3b2d81f1cc13e440c6d Nov 03 16:34:51.588818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.588766 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" podStartSLOduration=1.588750975 podStartE2EDuration="1.588750975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:51.587153013 +0000 UTC m=+1186.920366579" watchObservedRunningTime="2026-11-03 16:34:51.588750975 +0000 UTC m=+1186.921964543" Nov 03 16:34:51.637304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:51.637245 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podStartSLOduration=1.637229365 podStartE2EDuration="1.637229365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:51.636007233 +0000 UTC m=+1186.969220803" watchObservedRunningTime="2026-11-03 16:34:51.637229365 +0000 UTC m=+1186.970442931" Nov 03 16:34:52.254148 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:52.254103 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:34:52.532493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.532387 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:52.538020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.537988 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:52.554981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.554946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" event={"ID":"6a578f35-4d6b-4672-b34e-cc18a37c7cea","Type":"ContainerStarted","Data":"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9"} Nov 03 16:34:52.555150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.555000 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" event={"ID":"6a578f35-4d6b-4672-b34e-cc18a37c7cea","Type":"ContainerStarted","Data":"eefdf6f69275e32b6497d6c9eeac1b4d6976fb0d8b1cc3b2d81f1cc13e440c6d"} Nov 03 16:34:52.555150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.555026 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:34:52.555303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.555283 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:52.556121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.556103 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:34:52.600775 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.600740 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 16:34:52.604641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.604618 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.614626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.614593 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 16:34:52.620087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.619957 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3a2fa2\"" Nov 03 16:34:52.620087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.620023 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3a2fa2-dockercfg-mcgpb\"" Nov 03 16:34:52.630732 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.630678 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" podStartSLOduration=2.630663633 podStartE2EDuration="2.630663633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:52.628858007 +0000 UTC m=+1187.962071576" watchObservedRunningTime="2026-11-03 16:34:52.630663633 +0000 UTC m=+1187.963877250" Nov 03 16:34:52.695405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.695373 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 16:34:52.699302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.699267 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.700228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.700195 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.700358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.700249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw4h\" (UniqueName: \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.700417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.700402 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.700730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.700695 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbst\" (UniqueName: \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.720080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.720048 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 16:34:52.806618 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.806497 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbst\" (UniqueName: \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.808479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.808413 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.808479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.808478 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw4h\" (UniqueName: \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.808715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.808596 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.809610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.809581 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.810614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.810591 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.835374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.835323 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbst\" (UniqueName: \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\") pod \"payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:52.842967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.842934 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw4h\" (UniqueName: \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\") pod \"payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:52.920193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:52.919779 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:53.016193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.016153 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:53.222225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.222131 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 16:34:53.223824 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:53.223314 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2feca8_6a16_4bc0_bfa6_03800266de1e.slice/crio-14231c59a3e11aebd45db079d3dad3990b018192e4d88956ab6859d532e976cc WatchSource:0}: Error finding container 14231c59a3e11aebd45db079d3dad3990b018192e4d88956ab6859d532e976cc: Status 404 returned error can't find the container with id 14231c59a3e11aebd45db079d3dad3990b018192e4d88956ab6859d532e976cc Nov 03 16:34:53.322562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.322519 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 16:34:53.322954 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:53.322673 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d0c403_83f2_49b3_94b3_23f05c1ef11c.slice/crio-bb2e7eeed8a3a09f63da5601d21dd2b750f64490f302fb69a50e254222a60522 WatchSource:0}: Error finding container bb2e7eeed8a3a09f63da5601d21dd2b750f64490f302fb69a50e254222a60522: Status 404 returned error can't find the container with id bb2e7eeed8a3a09f63da5601d21dd2b750f64490f302fb69a50e254222a60522 Nov 03 16:34:53.511127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.511084 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 16:34:53.514872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.514853 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.535521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.535488 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-460104-dockercfg-zcwn2\"" Nov 03 16:34:53.541397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.541361 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 16:34:53.563140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.563109 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-460104\"" Nov 03 16:34:53.567049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.566716 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" event={"ID":"ca2feca8-6a16-4bc0-bfa6-03800266de1e","Type":"ContainerStarted","Data":"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58"} Nov 03 16:34:53.567049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.566822 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" event={"ID":"ca2feca8-6a16-4bc0-bfa6-03800266de1e","Type":"ContainerStarted","Data":"14231c59a3e11aebd45db079d3dad3990b018192e4d88956ab6859d532e976cc"} Nov 03 16:34:53.567049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.566877 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:34:53.569410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.569372 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" event={"ID":"f3d0c403-83f2-49b3-94b3-23f05c1ef11c","Type":"ContainerStarted","Data":"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd"} Nov 03 16:34:53.569598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.569434 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" event={"ID":"f3d0c403-83f2-49b3-94b3-23f05c1ef11c","Type":"ContainerStarted","Data":"bb2e7eeed8a3a09f63da5601d21dd2b750f64490f302fb69a50e254222a60522"} Nov 03 16:34:53.569598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.569521 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:34:53.627915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.627786 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.628543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.628515 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvs48\" (UniqueName: \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.649082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.649026 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" podStartSLOduration=1.649011281 podStartE2EDuration="1.649011281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:53.647680803 +0000 UTC m=+1188.980894371" watchObservedRunningTime="2026-11-03 16:34:53.649011281 +0000 UTC m=+1188.982224849" Nov 03 16:34:53.652191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.652146 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 16:34:53.655954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.655935 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.678375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.678344 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 16:34:53.709995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.709938 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" podStartSLOduration=1.709921253 podStartE2EDuration="1.709921253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:53.708597907 +0000 UTC m=+1189.041811477" watchObservedRunningTime="2026-11-03 16:34:53.709921253 +0000 UTC m=+1189.043134822" Nov 03 16:34:53.729767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.729722 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shnj\" (UniqueName: \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.729949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.729778 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.729949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.729904 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.730083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.729977 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvs48\" (UniqueName: \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.730565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.730534 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.757705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.757669 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvs48\" (UniqueName: \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\") pod \"payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.833797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.830278 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:53.833797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.831361 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.833797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.831647 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shnj\" (UniqueName: \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.833797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.832326 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.856388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.856353 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shnj\" (UniqueName: \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\") pod \"payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:53.970141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:53.970106 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:54.137992 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:54.137962 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95372900_df8f_4882_b20d_f589727dc7b4.slice/crio-e5e65d0959b3bdb35604df7add15466bc3f58e591bf66fd344b1e0d41da0c979 WatchSource:0}: Error finding container e5e65d0959b3bdb35604df7add15466bc3f58e591bf66fd344b1e0d41da0c979: Status 404 returned error can't find the container with id e5e65d0959b3bdb35604df7add15466bc3f58e591bf66fd344b1e0d41da0c979 Nov 03 16:34:54.138490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.138461 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 16:34:54.229262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.229230 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:54.250871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.250838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.250897 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251050 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251081 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251216 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251272 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251269 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:54.251410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251301 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251325 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:54.251410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251348 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251394 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\") pod \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\" (UID: \"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73\") " Nov 03 16:34:54.251639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.251520 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:54.252221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.252143 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.252221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.252167 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.252221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.252200 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.252221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.252145 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:54.252221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.252191 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:54.253999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.253957 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:34:54.254235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.254195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:54.254235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.254216 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "kube-api-access-jjcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:54.255108 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.255082 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token" pod "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" (UID: "5ec3b3d6-f6ad-4259-ad52-9b56686ffc73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:54.302487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.302453 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 16:34:54.304097 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:54.304064 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbe941a_2ad2_4c94_9d3a_dad871031c3b.slice/crio-35f05874923752053875279d0c7a9c474590a125b5f7b897b5555b2362b40539 WatchSource:0}: Error finding container 35f05874923752053875279d0c7a9c474590a125b5f7b897b5555b2362b40539: Status 404 returned error can't find the container with id 35f05874923752053875279d0c7a9c474590a125b5f7b897b5555b2362b40539 Nov 03 16:34:54.353843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353812 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.353843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353839 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjcgr\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-kube-api-access-jjcgr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.353843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353849 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.354259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353859 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.354259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353868 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.354259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.353877 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:34:54.577284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.577239 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" event={"ID":"3bbe941a-2ad2-4c94-9d3a-dad871031c3b","Type":"ContainerStarted","Data":"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be"} Nov 03 16:34:54.577548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.577308 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" event={"ID":"3bbe941a-2ad2-4c94-9d3a-dad871031c3b","Type":"ContainerStarted","Data":"35f05874923752053875279d0c7a9c474590a125b5f7b897b5555b2362b40539"} Nov 03 16:34:54.577548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.577438 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:34:54.578919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.578892 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" event={"ID":"95372900-df8f-4882-b20d-f589727dc7b4","Type":"ContainerStarted","Data":"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a"} Nov 03 16:34:54.579033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.578940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" event={"ID":"95372900-df8f-4882-b20d-f589727dc7b4","Type":"ContainerStarted","Data":"e5e65d0959b3bdb35604df7add15466bc3f58e591bf66fd344b1e0d41da0c979"} Nov 03 16:34:54.579033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.578971 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:34:54.580471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.580416 6001 generic.go:356] "Generic (PLEG): container finished" podID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerID="25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc" exitCode=0 Nov 03 16:34:54.580471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.580462 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" Nov 03 16:34:54.580638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.580498 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerDied","Data":"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc"} Nov 03 16:34:54.580638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.580537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq" event={"ID":"5ec3b3d6-f6ad-4259-ad52-9b56686ffc73","Type":"ContainerDied","Data":"47ec4dd4ee1e0de34b90b9c1ccd6a0f1f1c7f7bec42950987898ac0e2fc42a1f"} Nov 03 16:34:54.580638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.580567 6001 scope.go:122] "RemoveContainer" containerID="25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc" Nov 03 16:34:54.590931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.590912 6001 scope.go:122] "RemoveContainer" containerID="c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b" Nov 03 16:34:54.599955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.599930 6001 scope.go:122] "RemoveContainer" containerID="25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc" Nov 03 16:34:54.600276 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:54.600253 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc\": container with ID starting with 25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc not found: ID does not exist" containerID="25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc" Nov 03 16:34:54.600349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.600291 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc"} err="failed to get container status \"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc\": rpc error: code = NotFound desc = could not find container \"25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc\": container with ID starting with 25a79c0a38749addea17ac3cda25aeb4f98d84ca02dfc6dcb0a16db2a97443cc not found: ID does not exist" Nov 03 16:34:54.600349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.600317 6001 scope.go:122] "RemoveContainer" containerID="c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b" Nov 03 16:34:54.600645 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:34:54.600624 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b\": container with ID starting with c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b not found: ID does not exist" containerID="c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b" Nov 03 16:34:54.600696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.600651 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b"} err="failed to get container status \"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b\": rpc error: code = NotFound desc = could not find container \"c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b\": container with ID starting with c08d9752f758a198e17f6e2c735d21fdb024161ac8828554ea66fa5b9f1d709b not found: ID does not exist" Nov 03 16:34:54.634844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.634789 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" podStartSLOduration=1.6347737439999999 podStartE2EDuration="1.634773744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:54.634142696 +0000 UTC m=+1189.967356266" watchObservedRunningTime="2026-11-03 16:34:54.634773744 +0000 UTC m=+1189.967987313" Nov 03 16:34:54.686621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.686587 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:54.691696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.691655 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b5770a0e-openshift-default-6c6895dc5-mwdxq"] Nov 03 16:34:54.739613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.739558 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" podStartSLOduration=1.7395425119999999 podStartE2EDuration="1.739542512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:54.737512023 +0000 UTC m=+1190.070725591" watchObservedRunningTime="2026-11-03 16:34:54.739542512 +0000 UTC m=+1190.072756080" Nov 03 16:34:54.788306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:54.788273 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" path="/var/lib/kubelet/pods/5ec3b3d6-f6ad-4259-ad52-9b56686ffc73/volumes" Nov 03 16:34:57.166469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:57.166405 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh"] Nov 03 16:34:57.177857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:57.177824 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4d5492de-s5wgh"] Nov 03 16:34:58.019237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019203 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 16:34:58.019746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019718 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.019746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019747 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.019841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019762 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.019841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019767 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.019903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.019857 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.024842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.024820 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.041641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.041611 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-71d7b4\"" Nov 03 16:34:58.043356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.043332 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 16:34:58.053805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.053778 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-71d7b4-dockercfg-9b9qk\"" Nov 03 16:34:58.124986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.124940 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6nz\" (UniqueName: \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.125187 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.125004 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.139049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.139020 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 16:34:58.139636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.139622 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec3b3d6-f6ad-4259-ad52-9b56686ffc73" containerName="istio-proxy" Nov 03 16:34:58.143076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.143051 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.167112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.167073 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 16:34:58.225907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.225876 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6nz\" (UniqueName: \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.226074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.225932 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.226074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.226004 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvt78\" (UniqueName: \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.226074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.226031 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.226869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.226847 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.250550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.250518 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6nz\" (UniqueName: \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\") pod \"payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.327567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.327497 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvt78\" (UniqueName: \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.327567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.327533 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.328195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.328177 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.338564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.338534 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:58.351787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.351762 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvt78\" (UniqueName: \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\") pod \"payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.466836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.466797 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:58.595274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.593382 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 16:34:58.598618 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.598589 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.621942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.621915 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-491905-openshift-default-dockercfg-dc6wb\"" Nov 03 16:34:58.628463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.628333 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 16:34:58.631221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631190 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631339 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631496 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631547 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631569 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631655 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.631724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.631720 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgh7\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.650920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.650890 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 16:34:58.653610 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:58.653577 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f153cb0_a64a_460f_8510_f3680f0d9b87.slice/crio-b4d7f7b910a9911bd01b328c9c0d9a898bd0a1ea0b63d78294f27b22196c6ff6 WatchSource:0}: Error finding container b4d7f7b910a9911bd01b328c9c0d9a898bd0a1ea0b63d78294f27b22196c6ff6: Status 404 returned error can't find the container with id b4d7f7b910a9911bd01b328c9c0d9a898bd0a1ea0b63d78294f27b22196c6ff6 Nov 03 16:34:58.733405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733380 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733509 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733552 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733596 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733659 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgh7\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733879 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.733906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733894 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733930 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.733976 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.734313 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.734497 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.734599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.734529 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.736153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.736128 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.736413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.736388 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.758129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.758098 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgh7\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.758227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.758140 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:58.768974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.768948 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 16:34:58.769737 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:58.769714 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1745b37d_cab1_4761_81ff_83656d6490a6.slice/crio-d768eeabcdb3c745343f34b99b7ff87eb616bbc70aa0efba7b4a521b0f42a15b WatchSource:0}: Error finding container d768eeabcdb3c745343f34b99b7ff87eb616bbc70aa0efba7b4a521b0f42a15b: Status 404 returned error can't find the container with id d768eeabcdb3c745343f34b99b7ff87eb616bbc70aa0efba7b4a521b0f42a15b Nov 03 16:34:58.789593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.789563 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e89601a-ee5a-4b69-9c86-b94f014805a2" path="/var/lib/kubelet/pods/5e89601a-ee5a-4b69-9c86-b94f014805a2/volumes" Nov 03 16:34:58.913119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:58.913033 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:34:59.233234 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.233201 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 16:34:59.234215 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:34:59.234191 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb03d546_7b51_4c71_be04_8791f7985d18.slice/crio-2271af374cd1e3e4a9db9c77506cc9880a2454d043e51a033c2eec4ea0d00c6f WatchSource:0}: Error finding container 2271af374cd1e3e4a9db9c77506cc9880a2454d043e51a033c2eec4ea0d00c6f: Status 404 returned error can't find the container with id 2271af374cd1e3e4a9db9c77506cc9880a2454d043e51a033c2eec4ea0d00c6f Nov 03 16:34:59.236572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.236538 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:59.236688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.236619 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:59.236688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.236661 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:34:59.612253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.612155 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" event={"ID":"6f153cb0-a64a-460f-8510-f3680f0d9b87","Type":"ContainerStarted","Data":"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f"} Nov 03 16:34:59.612253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.612224 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" event={"ID":"6f153cb0-a64a-460f-8510-f3680f0d9b87","Type":"ContainerStarted","Data":"b4d7f7b910a9911bd01b328c9c0d9a898bd0a1ea0b63d78294f27b22196c6ff6"} Nov 03 16:34:59.612492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.612258 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:34:59.613749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.613725 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerStarted","Data":"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31"} Nov 03 16:34:59.613841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.613767 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerStarted","Data":"2271af374cd1e3e4a9db9c77506cc9880a2454d043e51a033c2eec4ea0d00c6f"} Nov 03 16:34:59.615616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.615339 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" event={"ID":"1745b37d-cab1-4761-81ff-83656d6490a6","Type":"ContainerStarted","Data":"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e"} Nov 03 16:34:59.615777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.615759 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" event={"ID":"1745b37d-cab1-4761-81ff-83656d6490a6","Type":"ContainerStarted","Data":"d768eeabcdb3c745343f34b99b7ff87eb616bbc70aa0efba7b4a521b0f42a15b"} Nov 03 16:34:59.615986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.615972 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:34:59.657582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.657529 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" podStartSLOduration=2.6575147230000002 podStartE2EDuration="2.657514723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:59.655096623 +0000 UTC m=+1194.988310192" watchObservedRunningTime="2026-11-03 16:34:59.657514723 +0000 UTC m=+1194.990728292" Nov 03 16:34:59.699484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.699421 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" podStartSLOduration=1.699404489 podStartE2EDuration="1.699404489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:59.697996522 +0000 UTC m=+1195.031210091" watchObservedRunningTime="2026-11-03 16:34:59.699404489 +0000 UTC m=+1195.032618058" Nov 03 16:34:59.746344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.746293 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podStartSLOduration=1.746279025 podStartE2EDuration="1.746279025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:59.744708026 +0000 UTC m=+1195.077921594" watchObservedRunningTime="2026-11-03 16:34:59.746279025 +0000 UTC m=+1195.079492593" Nov 03 16:34:59.914120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:34:59.914030 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:00.919795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:00.919765 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:01.628096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:01.628066 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:01.629328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:01.629304 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:02.560548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:02.560510 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 16:35:03.573478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:03.573415 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 16:35:04.585824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:04.585787 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 16:35:04.587166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:04.587138 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 16:35:04.926457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:04.926358 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:35:04.930406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:04.930379 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:35:05.591404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:05.591377 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 16:35:05.591879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:05.591465 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 16:35:10.625016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:10.624985 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 16:35:10.625464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:10.625094 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 16:35:16.086052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.086009 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 16:35:16.104736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.104708 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.122571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.122551 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"kube-root-ca.crt\"" Nov 03 16:35:16.124853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.124832 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 16:35:16.135749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.135724 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"default-dockercfg-zzp5s\"" Nov 03 16:35:16.135749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.135739 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"openshift-service-ca.crt\"" Nov 03 16:35:16.135884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.135746 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"simulator-w3-3a2fa2-kserve-self-signed-certs\"" Nov 03 16:35:16.162284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.161772 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.174025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.162331 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.174025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.162504 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4v2p\" (UniqueName: \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.174025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.162557 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.174025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.162592 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.174025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.162633 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.263695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263656 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.263857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263708 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.263857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263781 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4v2p\" (UniqueName: \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.263857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263804 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.263857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263827 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.264083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.263864 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.264171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.264149 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.264393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.264366 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.264483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.264393 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.266259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.266233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.266611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.266589 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.288387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.288359 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4v2p\" (UniqueName: \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\") pod \"simulator-w3-3a2fa2-kserve-755db874bc-p5zl2\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.424917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.424818 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:16.726557 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.726520 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 16:35:16.727578 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:35:16.727553 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ffda108_c908_449e_a944_ff58dfc95125.slice/crio-3fd19143a11f35cdcbffb9523fa0d72c3f351d4d6fd702903fc1c82b7b885a03 WatchSource:0}: Error finding container 3fd19143a11f35cdcbffb9523fa0d72c3f351d4d6fd702903fc1c82b7b885a03: Status 404 returned error can't find the container with id 3fd19143a11f35cdcbffb9523fa0d72c3f351d4d6fd702903fc1c82b7b885a03 Nov 03 16:35:16.733957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.733933 6001 generic.go:356] "Generic (PLEG): container finished" podID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerID="9050790ed792e206c05e678d31600a34e7312d57810bc9906548682f6681f029" exitCode=0 Nov 03 16:35:16.734058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.734006 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerDied","Data":"9050790ed792e206c05e678d31600a34e7312d57810bc9906548682f6681f029"} Nov 03 16:35:16.734476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.734434 6001 scope.go:122] "RemoveContainer" containerID="9050790ed792e206c05e678d31600a34e7312d57810bc9906548682f6681f029" Nov 03 16:35:16.737696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.737669 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:16.737772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.737739 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:16.737809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:16.737771 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:17.741729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:17.741677 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" event={"ID":"1ffda108-c908-449e-a944-ff58dfc95125","Type":"ContainerStarted","Data":"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d"} Nov 03 16:35:17.741729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:17.741735 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" event={"ID":"1ffda108-c908-449e-a944-ff58dfc95125","Type":"ContainerStarted","Data":"3fd19143a11f35cdcbffb9523fa0d72c3f351d4d6fd702903fc1c82b7b885a03"} Nov 03 16:35:17.755468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:17.751960 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerStarted","Data":"340a2683f45610236fc49654a1d23a7a782a2c82229d04fb0391409adaa3a4a6"} Nov 03 16:35:17.802262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:17.796928 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podStartSLOduration=1.796907807 podStartE2EDuration="1.796907807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:17.790013656 +0000 UTC m=+1213.123227228" watchObservedRunningTime="2026-11-03 16:35:17.796907807 +0000 UTC m=+1213.130121377" Nov 03 16:35:18.425937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:18.425899 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:18.532527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:18.532492 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:35:18.533763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:18.533738 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:35:18.533872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:18.533799 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:35:19.532152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:19.532115 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:35:19.532572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:19.532171 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:35:20.532752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:20.532710 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:35:20.533159 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:20.532772 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:35:21.532629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:21.532593 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:35:21.532821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:21.532674 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:35:22.308113 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:35:22.308061 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:35:22.532196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:22.532157 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 16:35:22.532368 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:22.532223 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 16:35:22.562565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:22.561681 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:35:23.539172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:23.539136 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:35:23.541106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:23.541077 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 16:35:26.426012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:26.425950 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:26.438872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:26.438837 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:26.868872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:26.868839 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:35:30.861031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.860939 6001 generic.go:356] "Generic (PLEG): container finished" podID="db03d546-7b51-4c71-be04-8791f7985d18" containerID="8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31" exitCode=0 Nov 03 16:35:30.861539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.861024 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerDied","Data":"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31"} Nov 03 16:35:30.861624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.861543 6001 scope.go:122] "RemoveContainer" containerID="8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31" Nov 03 16:35:30.862979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.862953 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:30.863053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.863020 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:30.863102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:30.863051 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:35:31.628460 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:31.628407 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:31.875777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:31.875733 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerStarted","Data":"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117"} Nov 03 16:35:31.914416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:31.914262 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:31.915691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:31.915658 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:31.915868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:31.915723 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:32.914069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:32.914031 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:32.914497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:32.914099 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:33.914221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:33.914186 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:33.914664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:33.914235 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:34.913655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:34.913621 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:34.913851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:34.913677 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:35.913833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:35.913799 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:35.914254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:35.913857 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:36.913846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:36.913798 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Nov 03 16:35:36.914375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:36.913863 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Nov 03 16:35:37.918198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:37.918166 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:37.918674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:37.918599 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:37.919410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:37.919391 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 16:35:39.032335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.032300 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 16:35:39.037003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.036976 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.058284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.058250 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 16:35:39.069108 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.069077 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"kube-root-ca.crt\"" Nov 03 16:35:39.069320 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.069078 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"openshift-service-ca.crt\"" Nov 03 16:35:39.069535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.069146 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"default-dockercfg-m22tm\"" Nov 03 16:35:39.078914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.078889 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"simulator-w1-491905-kserve-self-signed-certs\"" Nov 03 16:35:39.107863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.107817 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.107863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.107868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.107863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.107886 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rggm\" (UniqueName: \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.108195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.108007 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.108195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.108089 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.108195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.108141 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209100 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209138 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209163 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rggm\" (UniqueName: \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209201 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209261 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209335 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209606 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209618 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.209800 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.209754 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.211751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.211725 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.211947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.211924 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.264138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.264095 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rggm\" (UniqueName: \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\") pod \"simulator-w1-491905-kserve-7bc7dc8b86-5j5bc\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.351136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.351047 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:39.672998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.672969 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 16:35:39.673741 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:35:39.673717 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d725e50_1018_495a_b24d_42e60e376e61.slice/crio-10b3906c2b918606877140585a8d1be551c1cf3a3c22c18c6a1a665069f260e2 WatchSource:0}: Error finding container 10b3906c2b918606877140585a8d1be551c1cf3a3c22c18c6a1a665069f260e2: Status 404 returned error can't find the container with id 10b3906c2b918606877140585a8d1be551c1cf3a3c22c18c6a1a665069f260e2 Nov 03 16:35:39.932825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.932731 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" event={"ID":"7d725e50-1018-495a-b24d-42e60e376e61","Type":"ContainerStarted","Data":"4e18f543f0d987cae7499a0f55448416f25c2bd098aa2fe7010237104ba0d248"} Nov 03 16:35:39.932825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.932782 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" event={"ID":"7d725e50-1018-495a-b24d-42e60e376e61","Type":"ContainerStarted","Data":"10b3906c2b918606877140585a8d1be551c1cf3a3c22c18c6a1a665069f260e2"} Nov 03 16:35:39.980793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:39.980741 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podStartSLOduration=0.980726146 podStartE2EDuration="980.726146ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:39.979474204 +0000 UTC m=+1235.312687773" watchObservedRunningTime="2026-11-03 16:35:39.980726146 +0000 UTC m=+1235.313939715" Nov 03 16:35:40.351276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:40.351224 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:49.351779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:49.351731 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:49.364852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:49.364819 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:50.007894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:50.007863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:35:52.251587 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:35:52.251545 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:35:54.440968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.440922 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 16:35:54.447257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.447226 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.461010 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.460981 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 16:35:54.471977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.471861 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"default-dockercfg-2xrq7\"" Nov 03 16:35:54.471977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.471876 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"premium-w4-460104-kserve-self-signed-certs\"" Nov 03 16:35:54.475726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.475693 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"openshift-service-ca.crt\"" Nov 03 16:35:54.475726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.475720 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"kube-root-ca.crt\"" Nov 03 16:35:54.598127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598079 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.598309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598213 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.598360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftqgk\" (UniqueName: \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.598360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598352 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.598481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598374 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.598596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.598576 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.699934 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700002 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700054 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftqgk\" (UniqueName: \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700093 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700122 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700200 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700583 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700597 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.700727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.700679 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.702951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.702927 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.703355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.703331 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.724759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.724728 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftqgk\" (UniqueName: \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\") pod \"premium-w4-460104-kserve-5f5775c9c-cxfc6\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:54.762061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:54.762010 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:35:55.075600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:55.075568 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 16:35:55.078249 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:35:55.078216 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52a28929_28b8_42a3_b06a_a2a5851a8847.slice/crio-c41bf3fa05bb5069406d00467f3e8312d68553407d7ee6bd1e78039780178d46 WatchSource:0}: Error finding container c41bf3fa05bb5069406d00467f3e8312d68553407d7ee6bd1e78039780178d46: Status 404 returned error can't find the container with id c41bf3fa05bb5069406d00467f3e8312d68553407d7ee6bd1e78039780178d46 Nov 03 16:35:56.036238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:56.036199 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" event={"ID":"52a28929-28b8-42a3-b06a-a2a5851a8847","Type":"ContainerStarted","Data":"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e"} Nov 03 16:35:56.036238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:56.036247 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" event={"ID":"52a28929-28b8-42a3-b06a-a2a5851a8847","Type":"ContainerStarted","Data":"c41bf3fa05bb5069406d00467f3e8312d68553407d7ee6bd1e78039780178d46"} Nov 03 16:35:56.100968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:56.100895 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podStartSLOduration=2.100871289 podStartE2EDuration="2.100871289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:56.097818216 +0000 UTC m=+1251.431031787" watchObservedRunningTime="2026-11-03 16:35:56.100871289 +0000 UTC m=+1251.434084860" Nov 03 16:35:56.762755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:35:56.762723 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:36:04.763168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:04.763132 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:36:04.775834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:04.775809 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:36:05.104633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:05.104538 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:36:13.068922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.068882 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799"] Nov 03 16:36:13.074025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.074005 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.104677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.104639 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:13.104914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.104680 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:13.104914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.104680 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:13.104914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.104681 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:36:13.107037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.107012 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799"] Nov 03 16:36:13.142997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.142912 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.143155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.143049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgtkv\" (UniqueName: \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.143155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.143129 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.244407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.244365 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.244607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.244485 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.244607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.244551 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgtkv\" (UniqueName: \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.245055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.245027 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.246964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.246935 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.269375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.269338 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgtkv\" (UniqueName: \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\") pod \"maas-api-revoke-sub-595574b5-8k799\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.387984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.387945 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:13.690314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:13.690284 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799"] Nov 03 16:36:13.690886 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:13.690858 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ecd2c1_577b_47a9_afa9_e48cb0118122.slice/crio-a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c WatchSource:0}: Error finding container a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c: Status 404 returned error can't find the container with id a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c Nov 03 16:36:14.154372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:14.154331 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" event={"ID":"b4ecd2c1-577b-47a9-afa9-e48cb0118122","Type":"ContainerStarted","Data":"443b63832516dcf0685bfc0957b43b6eb757186af618d6f2c00aec9844b9d081"} Nov 03 16:36:14.154784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:14.154383 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" event={"ID":"b4ecd2c1-577b-47a9-afa9-e48cb0118122","Type":"ContainerStarted","Data":"a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c"} Nov 03 16:36:14.198582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:14.198513 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" podStartSLOduration=1.198497798 podStartE2EDuration="1.198497798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:14.197030149 +0000 UTC m=+1269.530243715" watchObservedRunningTime="2026-11-03 16:36:14.198497798 +0000 UTC m=+1269.531711367" Nov 03 16:36:15.161751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:15.161710 6001 generic.go:356] "Generic (PLEG): container finished" podID="b4ecd2c1-577b-47a9-afa9-e48cb0118122" containerID="443b63832516dcf0685bfc0957b43b6eb757186af618d6f2c00aec9844b9d081" exitCode=0 Nov 03 16:36:15.162181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:15.161797 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" event={"ID":"b4ecd2c1-577b-47a9-afa9-e48cb0118122","Type":"ContainerDied","Data":"443b63832516dcf0685bfc0957b43b6eb757186af618d6f2c00aec9844b9d081"} Nov 03 16:36:16.334485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.334437 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:16.382194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.382153 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\") pod \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " Nov 03 16:36:16.382382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.382265 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\" (UniqueName: \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\") pod \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " Nov 03 16:36:16.382382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.382297 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\") pod \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\" (UID: \"b4ecd2c1-577b-47a9-afa9-e48cb0118122\") " Nov 03 16:36:16.382745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.382714 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca" pod "b4ecd2c1-577b-47a9-afa9-e48cb0118122" (UID: "b4ecd2c1-577b-47a9-afa9-e48cb0118122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:16.382899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.382882 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:16.384458 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.384417 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca" pod "b4ecd2c1-577b-47a9-afa9-e48cb0118122" (UID: "b4ecd2c1-577b-47a9-afa9-e48cb0118122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:16.384627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.384609 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv" pod "b4ecd2c1-577b-47a9-afa9-e48cb0118122" (UID: "b4ecd2c1-577b-47a9-afa9-e48cb0118122"). InnerVolumeSpecName "kube-api-access-qgtkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:16.484132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.484090 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ecd2c1-577b-47a9-afa9-e48cb0118122-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:16.484328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:16.484126 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgtkv\" (UniqueName: \"kubernetes.io/projected/b4ecd2c1-577b-47a9-afa9-e48cb0118122-kube-api-access-qgtkv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:17.175200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:17.175110 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" Nov 03 16:36:17.175200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:17.175140 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799" event={"ID":"b4ecd2c1-577b-47a9-afa9-e48cb0118122","Type":"ContainerDied","Data":"a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c"} Nov 03 16:36:17.175200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:17.175183 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e082377d45e2bbe5c8f9f33a0f2f6680892f429dafafa6314ed238399d1b9c" Nov 03 16:36:22.249553 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:36:22.249517 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:36:28.313410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.313365 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt"] Nov 03 16:36:28.314155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.314131 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b4ecd2c1-577b-47a9-afa9-e48cb0118122" containerName="revoke-keys" Nov 03 16:36:28.314236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.314156 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ecd2c1-577b-47a9-afa9-e48cb0118122" containerName="revoke-keys" Nov 03 16:36:28.314306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.314293 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ecd2c1-577b-47a9-afa9-e48cb0118122" containerName="revoke-keys" Nov 03 16:36:28.317980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.317956 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.333607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.333582 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:28.346278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.346253 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:28.346278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.346253 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:28.346484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.346303 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:36:28.355727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.355650 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt"] Nov 03 16:36:28.432375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.432334 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.432375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.432379 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.432641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.432497 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9f6\" (UniqueName: \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.533259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.533218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9f6\" (UniqueName: \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.533504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.533378 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.533504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.533407 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.534080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.534054 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.535844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.535816 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.557698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.557655 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9f6\" (UniqueName: \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\") pod \"maas-api-revoke-sub-e44815ef-gj8vt\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.632041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.631953 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:28.939214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:28.939182 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt"] Nov 03 16:36:28.940220 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:28.940196 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f908ffb_6de3_4199_a9ab_1cc88b7524f3.slice/crio-e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7 WatchSource:0}: Error finding container e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7: Status 404 returned error can't find the container with id e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7 Nov 03 16:36:29.257866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:29.257831 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" event={"ID":"9f908ffb-6de3-4199-a9ab-1cc88b7524f3","Type":"ContainerStarted","Data":"e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7"} Nov 03 16:36:30.266173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:30.266138 6001 generic.go:356] "Generic (PLEG): container finished" podID="9f908ffb-6de3-4199-a9ab-1cc88b7524f3" containerID="2a26d47f3544f891417be93bdc4d49a0ba8752a3e8d1bb9872bcd81e5430f803" exitCode=0 Nov 03 16:36:30.266627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:30.266224 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" event={"ID":"9f908ffb-6de3-4199-a9ab-1cc88b7524f3","Type":"ContainerDied","Data":"2a26d47f3544f891417be93bdc4d49a0ba8752a3e8d1bb9872bcd81e5430f803"} Nov 03 16:36:31.421948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.421921 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:31.577475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.577355 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\") pod \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " Nov 03 16:36:31.577475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.577415 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\" (UniqueName: \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\") pod \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " Nov 03 16:36:31.577722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.577570 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\") pod \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\" (UID: \"9f908ffb-6de3-4199-a9ab-1cc88b7524f3\") " Nov 03 16:36:31.577864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.577832 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca" pod "9f908ffb-6de3-4199-a9ab-1cc88b7524f3" (UID: "9f908ffb-6de3-4199-a9ab-1cc88b7524f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:31.579987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.579954 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca" pod "9f908ffb-6de3-4199-a9ab-1cc88b7524f3" (UID: "9f908ffb-6de3-4199-a9ab-1cc88b7524f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:31.579987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.579970 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6" pod "9f908ffb-6de3-4199-a9ab-1cc88b7524f3" (UID: "9f908ffb-6de3-4199-a9ab-1cc88b7524f3"). InnerVolumeSpecName "kube-api-access-nq9f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:31.666937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.666903 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 16:36:31.667499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.667478 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9f908ffb-6de3-4199-a9ab-1cc88b7524f3" containerName="revoke-keys" Nov 03 16:36:31.667499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.667497 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f908ffb-6de3-4199-a9ab-1cc88b7524f3" containerName="revoke-keys" Nov 03 16:36:31.667653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.667636 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f908ffb-6de3-4199-a9ab-1cc88b7524f3" containerName="revoke-keys" Nov 03 16:36:31.671252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.671228 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.680698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.679249 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:31.680698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.679281 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9f6\" (UniqueName: \"kubernetes.io/projected/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-kube-api-access-nq9f6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:31.680698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.679297 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f908ffb-6de3-4199-a9ab-1cc88b7524f3-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:31.689064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.689030 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-460104-openshift-default-dockercfg-4t76j\"" Nov 03 16:36:31.698499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.696843 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 16:36:31.780242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780202 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780254 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780362 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780408 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkpc\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780566 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780602 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780690 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780746 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.780797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.780783 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.881987 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882046 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882088 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882109 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkpc\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882252 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882288 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882378 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882669 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882763 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882783 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.882947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.882872 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.883391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.883363 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.883391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.883381 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.883556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.883367 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.884677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.884653 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.884763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.884749 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.906389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.906351 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.906573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.906486 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkpc\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:31.988630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:31.988587 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:32.279356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.279324 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" Nov 03 16:36:32.279356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.279353 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt" event={"ID":"9f908ffb-6de3-4199-a9ab-1cc88b7524f3","Type":"ContainerDied","Data":"e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7"} Nov 03 16:36:32.279599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.279404 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12af43569e7caa1e67276315a137dfef8ac537f35b4c5dc3ec6104c490ab0e7" Nov 03 16:36:32.304736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.304709 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 16:36:32.306629 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:32.306599 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b166fa_1f1d_4e0e_95d2_bb2953244fe3.slice/crio-8ad7169f9f7c165c5168c55cc3c29b56d08322a42528dce2d02b6cec0c8ee494 WatchSource:0}: Error finding container 8ad7169f9f7c165c5168c55cc3c29b56d08322a42528dce2d02b6cec0c8ee494: Status 404 returned error can't find the container with id 8ad7169f9f7c165c5168c55cc3c29b56d08322a42528dce2d02b6cec0c8ee494 Nov 03 16:36:32.309315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.309280 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:36:32.309404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.309354 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:36:32.309404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:32.309385 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:36:33.288360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.288324 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" event={"ID":"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3","Type":"ContainerStarted","Data":"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169"} Nov 03 16:36:33.288869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.288388 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" event={"ID":"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3","Type":"ContainerStarted","Data":"8ad7169f9f7c165c5168c55cc3c29b56d08322a42528dce2d02b6cec0c8ee494"} Nov 03 16:36:33.370387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.370319 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podStartSLOduration=2.37030014 podStartE2EDuration="2.37030014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:33.357172495 +0000 UTC m=+1288.690386065" watchObservedRunningTime="2026-11-03 16:36:33.37030014 +0000 UTC m=+1288.703513712" Nov 03 16:36:33.989361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.989312 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:33.991018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.990987 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:36:33.991146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:33.991063 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:36:34.989680 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:34.989639 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:36:34.990126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:34.989710 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:36:35.989354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:35.989315 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:36:35.989573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:35.989389 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:36:36.990033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:36.989995 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:36:36.990490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:36.990056 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:36:37.989278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:37.989235 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:36:37.989485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:37.989333 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 16:36:39.017027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:39.016986 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:39.018233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:39.018200 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:39.020519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:39.020494 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 16:36:49.911976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.911941 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7"] Nov 03 16:36:49.916245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.916223 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:49.935702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.935668 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:49.945247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.945215 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:49.945496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.945264 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:49.945496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.945330 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:36:49.947738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:49.947717 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7"] Nov 03 16:36:50.014759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.014719 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.014919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.014777 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snnwl\" (UniqueName: \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.014981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.014926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.116480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.116428 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.116708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.116503 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snnwl\" (UniqueName: \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.116708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.116623 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.117174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.117145 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.118975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.118954 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.140696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.140662 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snnwl\" (UniqueName: \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\") pod \"maas-api-revoke-sub-6760d390-rlgl7\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.230548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.230517 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:50.539646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:50.539612 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7"] Nov 03 16:36:50.539646 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:50.539630 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0f2e06_b502_40e2_bc60_1d3741e6dd3f.slice/crio-7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406 WatchSource:0}: Error finding container 7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406: Status 404 returned error can't find the container with id 7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406 Nov 03 16:36:51.417281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:51.417241 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" event={"ID":"df0f2e06-b502-40e2-bc60-1d3741e6dd3f","Type":"ContainerStarted","Data":"9f144061c5ee6043ce443feb34d6dbaf13e403b596d1856c7803f32c593f9e5c"} Nov 03 16:36:51.417776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:51.417293 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" event={"ID":"df0f2e06-b502-40e2-bc60-1d3741e6dd3f","Type":"ContainerStarted","Data":"7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406"} Nov 03 16:36:51.460393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:51.460342 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" podStartSLOduration=2.46032711 podStartE2EDuration="2.46032711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:51.459527519 +0000 UTC m=+1306.792741088" watchObservedRunningTime="2026-11-03 16:36:51.46032711 +0000 UTC m=+1306.793540678" Nov 03 16:36:52.252644 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:36:52.252606 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:36:52.424826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:52.424789 6001 generic.go:356] "Generic (PLEG): container finished" podID="df0f2e06-b502-40e2-bc60-1d3741e6dd3f" containerID="9f144061c5ee6043ce443feb34d6dbaf13e403b596d1856c7803f32c593f9e5c" exitCode=0 Nov 03 16:36:52.425201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:52.424877 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" event={"ID":"df0f2e06-b502-40e2-bc60-1d3741e6dd3f","Type":"ContainerDied","Data":"9f144061c5ee6043ce443feb34d6dbaf13e403b596d1856c7803f32c593f9e5c"} Nov 03 16:36:53.604559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.604535 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:53.660169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.660134 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\") pod \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " Nov 03 16:36:53.660371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.660192 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\") pod \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " Nov 03 16:36:53.660371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.660236 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\" (UniqueName: \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\") pod \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\" (UID: \"df0f2e06-b502-40e2-bc60-1d3741e6dd3f\") " Nov 03 16:36:53.660580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.660545 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca" pod "df0f2e06-b502-40e2-bc60-1d3741e6dd3f" (UID: "df0f2e06-b502-40e2-bc60-1d3741e6dd3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:53.660790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.660767 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:53.662492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.662463 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca" pod "df0f2e06-b502-40e2-bc60-1d3741e6dd3f" (UID: "df0f2e06-b502-40e2-bc60-1d3741e6dd3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:53.662492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.662464 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl" pod "df0f2e06-b502-40e2-bc60-1d3741e6dd3f" (UID: "df0f2e06-b502-40e2-bc60-1d3741e6dd3f"). InnerVolumeSpecName "kube-api-access-snnwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:53.761885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.761844 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:53.761885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:53.761882 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snnwl\" (UniqueName: \"kubernetes.io/projected/df0f2e06-b502-40e2-bc60-1d3741e6dd3f-kube-api-access-snnwl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:54.441331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:54.441296 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" Nov 03 16:36:54.441550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:54.441303 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7" event={"ID":"df0f2e06-b502-40e2-bc60-1d3741e6dd3f","Type":"ContainerDied","Data":"7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406"} Nov 03 16:36:54.441550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:54.441440 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cce5eea17f805f555f99378b3ff2e68fc76e7abf7b71f8b6bf7ad8ea21ef406" Nov 03 16:36:56.732727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.732691 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z"] Nov 03 16:36:56.733341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.733324 6001 cpu_manager.go:414] "containerMap: removing container" podUID="df0f2e06-b502-40e2-bc60-1d3741e6dd3f" containerName="revoke-keys" Nov 03 16:36:56.733385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.733342 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0f2e06-b502-40e2-bc60-1d3741e6dd3f" containerName="revoke-keys" Nov 03 16:36:56.733478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.733438 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0f2e06-b502-40e2-bc60-1d3741e6dd3f" containerName="revoke-keys" Nov 03 16:36:56.735997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.735969 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.755160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.755128 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:36:56.755310 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.755197 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:36:56.771743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.771711 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:36:56.771972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.771777 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:36:56.775643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.775608 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z"] Nov 03 16:36:56.799994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.799946 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvxm\" (UniqueName: \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.800200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.800080 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.800200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.800157 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.901733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.901259 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.901733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.901367 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.901733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.901504 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvxm\" (UniqueName: \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.903689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.902698 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.913378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.913340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:56.926870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:56.926836 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvxm\" (UniqueName: \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\") pod \"maas-api-revoke-sub-db7ee809-66k2z\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:57.050280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.050202 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:57.378852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.378817 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z"] Nov 03 16:36:57.379263 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:57.379237 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d71caa4_66a4_4435_bdbd_7ea4fc7d1487.slice/crio-d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4 WatchSource:0}: Error finding container d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4: Status 404 returned error can't find the container with id d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4 Nov 03 16:36:57.461740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.461702 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" event={"ID":"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487","Type":"ContainerStarted","Data":"d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4"} Nov 03 16:36:57.759798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.759773 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288"] Nov 03 16:36:57.762418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.762402 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:57.790464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.790417 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288"] Nov 03 16:36:57.914069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.913973 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:57.914249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.914108 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvpxp\" (UniqueName: \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:57.914249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:57.914164 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.015390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.015351 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.015609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.015476 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvpxp\" (UniqueName: \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.015609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.015531 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.016032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.016010 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.017902 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.017883 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.039731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.039702 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvpxp\" (UniqueName: \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\") pod \"maas-api-revoke-sub-da4ca32d-ch288\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.074728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.074692 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:36:58.373761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.373729 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288"] Nov 03 16:36:58.374015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:36:58.373992 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b2f62d_ac2e_44fa_864c_9b6e3be1fe5f.slice/crio-8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19 WatchSource:0}: Error finding container 8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19: Status 404 returned error can't find the container with id 8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19 Nov 03 16:36:58.469395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.469358 6001 generic.go:356] "Generic (PLEG): container finished" podID="0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" containerID="76dc337b66505f2e6daa2eb46bb0ff678cad55ca684271e3c4e9871de7b8d0ce" exitCode=0 Nov 03 16:36:58.469594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.469399 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" event={"ID":"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487","Type":"ContainerDied","Data":"76dc337b66505f2e6daa2eb46bb0ff678cad55ca684271e3c4e9871de7b8d0ce"} Nov 03 16:36:58.470901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:58.470876 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" event={"ID":"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f","Type":"ContainerStarted","Data":"8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19"} Nov 03 16:36:59.480132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.480094 6001 generic.go:356] "Generic (PLEG): container finished" podID="b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" containerID="80cf8c1e557d06f44bf19270a4247d796c645f121108654815ea891c02922669" exitCode=0 Nov 03 16:36:59.480561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.480183 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" event={"ID":"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f","Type":"ContainerDied","Data":"80cf8c1e557d06f44bf19270a4247d796c645f121108654815ea891c02922669"} Nov 03 16:36:59.650535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.650505 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:36:59.736679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.736599 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\" (UniqueName: \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\") pod \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " Nov 03 16:36:59.736846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.736693 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\") pod \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " Nov 03 16:36:59.736846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.736762 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\") pod \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\" (UID: \"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487\") " Nov 03 16:36:59.737078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.737049 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca" pod "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" (UID: "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:36:59.737398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.737374 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:59.739395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.739363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm" pod "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" (UID: "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487"). InnerVolumeSpecName "kube-api-access-zlvxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:36:59.739520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.739363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca" pod "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" (UID: "0d71caa4-66a4-4435-bdbd-7ea4fc7d1487"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:36:59.838430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.838397 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlvxm\" (UniqueName: \"kubernetes.io/projected/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-kube-api-access-zlvxm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:36:59.838430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:36:59.838427 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:00.141172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.141078 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp"] Nov 03 16:37:00.142048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.142023 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" containerName="revoke-keys" Nov 03 16:37:00.142184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.142048 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" containerName="revoke-keys" Nov 03 16:37:00.142377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.142360 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" containerName="revoke-keys" Nov 03 16:37:00.145521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.145494 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.163577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.163547 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp"] Nov 03 16:37:00.249675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.249614 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwfhs\" (UniqueName: \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.249884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.249772 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.249884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.249826 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.351637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.351596 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.351826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.351650 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.351875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.351827 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwfhs\" (UniqueName: \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.352378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.352357 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.354137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.354115 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.376804 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.376772 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwfhs\" (UniqueName: \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\") pod \"maas-api-revoke-sub-f2eebaf6-g5qvp\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.466694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.466600 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:00.488834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.488804 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" Nov 03 16:37:00.488834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.488817 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z" event={"ID":"0d71caa4-66a4-4435-bdbd-7ea4fc7d1487","Type":"ContainerDied","Data":"d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4"} Nov 03 16:37:00.489332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.488875 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a28fa09c77046ad61bac887138228a4f6c3a1c70f08306ab2cddb475fdefb4" Nov 03 16:37:00.632735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.632709 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:37:00.755950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.755917 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\" (UniqueName: \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\") pod \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " Nov 03 16:37:00.756272 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.756254 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\") pod \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " Nov 03 16:37:00.756351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.756312 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\") pod \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\" (UID: \"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f\") " Nov 03 16:37:00.756631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.756597 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca" pod "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" (UID: "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:00.756776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.756759 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:00.758324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.758300 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp" pod "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" (UID: "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f"). InnerVolumeSpecName "kube-api-access-kvpxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:00.758418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.758395 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca" pod "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" (UID: "b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:00.768914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.768887 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:37:00.769480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.769461 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" containerName="revoke-keys" Nov 03 16:37:00.769622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.769482 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" containerName="revoke-keys" Nov 03 16:37:00.769676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.769644 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" containerName="revoke-keys" Nov 03 16:37:00.772272 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.772254 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.774968 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:00.774942 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1c9128_ae78_445f_94d5_20620e5fd9e8.slice/crio-e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a WatchSource:0}: Error finding container e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a: Status 404 returned error can't find the container with id e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a Nov 03 16:37:00.775342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.775320 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp"] Nov 03 16:37:00.787965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.787937 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7519b7-openshift-default-dockercfg-dv7k2\"" Nov 03 16:37:00.795617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.795591 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:37:00.858500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858422 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858541 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858604 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ds8\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858685 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858730 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858839 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858882 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.858967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.859005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.858993 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:00.859005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.859003 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvpxp\" (UniqueName: \"kubernetes.io/projected/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f-kube-api-access-kvpxp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:00.960425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960392 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960474 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960505 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ds8\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960556 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960617 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960678 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960740 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960799 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.960835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.960828 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.961196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.961174 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.961456 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.961410 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.961544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.961479 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.961544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.961426 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.961621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.961551 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.962983 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.962961 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.963238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.963217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.984404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.984371 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:00.984601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:00.984585 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ds8\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:01.086109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.086078 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:01.401263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.401234 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:37:01.401263 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:01.401249 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb149bb_32f8_438c_842b_364c222becfc.slice/crio-11e90a9fe2a22f858ef7b740e47f48b4728c8c32413711347ae0a9ca7130912a WatchSource:0}: Error finding container 11e90a9fe2a22f858ef7b740e47f48b4728c8c32413711347ae0a9ca7130912a: Status 404 returned error can't find the container with id 11e90a9fe2a22f858ef7b740e47f48b4728c8c32413711347ae0a9ca7130912a Nov 03 16:37:01.403800 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.403767 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:01.403888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.403837 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:01.403888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.403867 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:01.497585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.497545 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" event={"ID":"fd1c9128-ae78-445f-94d5-20620e5fd9e8","Type":"ContainerStarted","Data":"a480a484bd241430cfdd130ea481066b86274faaf5c4d8cfdc4e548a357a25f8"} Nov 03 16:37:01.498030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.497612 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" event={"ID":"fd1c9128-ae78-445f-94d5-20620e5fd9e8","Type":"ContainerStarted","Data":"e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a"} Nov 03 16:37:01.499570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.499539 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" event={"ID":"8cb149bb-32f8-438c-842b-364c222becfc","Type":"ContainerStarted","Data":"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033"} Nov 03 16:37:01.499708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.499587 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" event={"ID":"8cb149bb-32f8-438c-842b-364c222becfc","Type":"ContainerStarted","Data":"11e90a9fe2a22f858ef7b740e47f48b4728c8c32413711347ae0a9ca7130912a"} Nov 03 16:37:01.501273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.501250 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" Nov 03 16:37:01.501403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.501280 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288" event={"ID":"b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f","Type":"ContainerDied","Data":"8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19"} Nov 03 16:37:01.501403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.501316 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e4517851a5fc73aa7569adab7af94222fb916bba52c8317fe0ab3286e12ab19" Nov 03 16:37:01.540201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.540139 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" podStartSLOduration=1.540125972 podStartE2EDuration="1.540125972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:01.539917212 +0000 UTC m=+1316.873130781" watchObservedRunningTime="2026-11-03 16:37:01.540125972 +0000 UTC m=+1316.873339541" Nov 03 16:37:01.610868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:01.610804 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podStartSLOduration=1.6107844999999998 podStartE2EDuration="1.6107845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:01.609013793 +0000 UTC m=+1316.942227363" watchObservedRunningTime="2026-11-03 16:37:01.6107845 +0000 UTC m=+1316.943998069" Nov 03 16:37:02.086323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:02.086279 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:02.088366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:02.088339 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:02.088526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:02.088389 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:02.513087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:02.513047 6001 generic.go:356] "Generic (PLEG): container finished" podID="fd1c9128-ae78-445f-94d5-20620e5fd9e8" containerID="a480a484bd241430cfdd130ea481066b86274faaf5c4d8cfdc4e548a357a25f8" exitCode=0 Nov 03 16:37:02.513598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:02.513199 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" event={"ID":"fd1c9128-ae78-445f-94d5-20620e5fd9e8","Type":"ContainerDied","Data":"a480a484bd241430cfdd130ea481066b86274faaf5c4d8cfdc4e548a357a25f8"} Nov 03 16:37:03.087019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.086977 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:03.087220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.087048 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:03.700853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.700822 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:03.897615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.897522 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\") pod \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " Nov 03 16:37:03.897615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.897568 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\") pod \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " Nov 03 16:37:03.897855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.897695 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\" (UniqueName: \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\") pod \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\" (UID: \"fd1c9128-ae78-445f-94d5-20620e5fd9e8\") " Nov 03 16:37:03.897920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.897894 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca" pod "fd1c9128-ae78-445f-94d5-20620e5fd9e8" (UID: "fd1c9128-ae78-445f-94d5-20620e5fd9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:03.898273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.898253 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:03.899788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.899761 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca" pod "fd1c9128-ae78-445f-94d5-20620e5fd9e8" (UID: "fd1c9128-ae78-445f-94d5-20620e5fd9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:03.899896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.899818 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs" pod "fd1c9128-ae78-445f-94d5-20620e5fd9e8" (UID: "fd1c9128-ae78-445f-94d5-20620e5fd9e8"). InnerVolumeSpecName "kube-api-access-bwfhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:03.999388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.999345 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwfhs\" (UniqueName: \"kubernetes.io/projected/fd1c9128-ae78-445f-94d5-20620e5fd9e8-kube-api-access-bwfhs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:03.999388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:03.999380 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd1c9128-ae78-445f-94d5-20620e5fd9e8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:04.086943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.086900 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:04.087115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.086958 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:04.441809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.441770 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt"] Nov 03 16:37:04.442417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.442401 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fd1c9128-ae78-445f-94d5-20620e5fd9e8" containerName="revoke-keys" Nov 03 16:37:04.442482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.442418 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1c9128-ae78-445f-94d5-20620e5fd9e8" containerName="revoke-keys" Nov 03 16:37:04.442561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.442544 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1c9128-ae78-445f-94d5-20620e5fd9e8" containerName="revoke-keys" Nov 03 16:37:04.469511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.462760 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.471154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.471121 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt"] Nov 03 16:37:04.526697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.526671 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" Nov 03 16:37:04.526836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.526708 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp" event={"ID":"fd1c9128-ae78-445f-94d5-20620e5fd9e8","Type":"ContainerDied","Data":"e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a"} Nov 03 16:37:04.526836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.526759 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4e57306edb467e30ff3ed0924cacbdc3e6794e1ce0483c04a89ae91dbd0856a" Nov 03 16:37:04.605800 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.605754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.605984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.605877 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj72v\" (UniqueName: \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.606060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.606043 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.707179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.707096 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.707583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.707209 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.707583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.707268 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj72v\" (UniqueName: \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.707944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.707925 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.709609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.709594 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.731671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.731646 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj72v\" (UniqueName: \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\") pod \"maas-api-revoke-sub-c548914b-z28lt\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:04.785955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:04.785922 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:05.087071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.087030 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:05.087263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.087106 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:05.099192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.099158 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt"] Nov 03 16:37:05.100512 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:05.100482 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9249befe_1afe_43e4_97b4_cb9ee45faac2.slice/crio-46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f WatchSource:0}: Error finding container 46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f: Status 404 returned error can't find the container with id 46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f Nov 03 16:37:05.534470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.534422 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" event={"ID":"9249befe-1afe-43e4-97b4-cb9ee45faac2","Type":"ContainerStarted","Data":"f4597fff32c0c307f78c1a2e900b6ef8752167e0bd545ceb0d5e83eb684e77d9"} Nov 03 16:37:05.534651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.534513 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" event={"ID":"9249befe-1afe-43e4-97b4-cb9ee45faac2","Type":"ContainerStarted","Data":"46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f"} Nov 03 16:37:05.585128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:05.585078 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" podStartSLOduration=1.585064358 podStartE2EDuration="1.585064358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:05.583723222 +0000 UTC m=+1320.916936793" watchObservedRunningTime="2026-11-03 16:37:05.585064358 +0000 UTC m=+1320.918277926" Nov 03 16:37:06.086718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:06.086678 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:06.087128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:06.086741 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:06.542488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:06.542436 6001 generic.go:356] "Generic (PLEG): container finished" podID="9249befe-1afe-43e4-97b4-cb9ee45faac2" containerID="f4597fff32c0c307f78c1a2e900b6ef8752167e0bd545ceb0d5e83eb684e77d9" exitCode=0 Nov 03 16:37:06.542488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:06.542476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" event={"ID":"9249befe-1afe-43e4-97b4-cb9ee45faac2","Type":"ContainerDied","Data":"f4597fff32c0c307f78c1a2e900b6ef8752167e0bd545ceb0d5e83eb684e77d9"} Nov 03 16:37:07.087073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.087042 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 16:37:07.087478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.087105 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 16:37:07.695655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.695624 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:07.744049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.744008 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\") pod \"9249befe-1afe-43e4-97b4-cb9ee45faac2\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " Nov 03 16:37:07.744223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.744081 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\" (UniqueName: \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\") pod \"9249befe-1afe-43e4-97b4-cb9ee45faac2\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " Nov 03 16:37:07.744223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.744134 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\") pod \"9249befe-1afe-43e4-97b4-cb9ee45faac2\" (UID: \"9249befe-1afe-43e4-97b4-cb9ee45faac2\") " Nov 03 16:37:07.744967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.744935 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca" pod "9249befe-1afe-43e4-97b4-cb9ee45faac2" (UID: "9249befe-1afe-43e4-97b4-cb9ee45faac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:07.747335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.747301 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca" pod "9249befe-1afe-43e4-97b4-cb9ee45faac2" (UID: "9249befe-1afe-43e4-97b4-cb9ee45faac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:07.748105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.748081 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v" pod "9249befe-1afe-43e4-97b4-cb9ee45faac2" (UID: "9249befe-1afe-43e4-97b4-cb9ee45faac2"). InnerVolumeSpecName "kube-api-access-rj72v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:07.846043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.845934 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:07.846043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.845971 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9249befe-1afe-43e4-97b4-cb9ee45faac2-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:07.846043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:07.845986 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj72v\" (UniqueName: \"kubernetes.io/projected/9249befe-1afe-43e4-97b4-cb9ee45faac2-kube-api-access-rj72v\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:08.163734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.163638 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:08.164208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.164028 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:08.165027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.165005 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:37:08.557323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.557294 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" Nov 03 16:37:08.557323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.557309 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt" event={"ID":"9249befe-1afe-43e4-97b4-cb9ee45faac2","Type":"ContainerDied","Data":"46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f"} Nov 03 16:37:08.557616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:08.557371 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46c486dbbd1a305dda9da436a6f388316134bbd72400162696e294373ae94b4f" Nov 03 16:37:10.551606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.551566 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck"] Nov 03 16:37:10.552168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.552153 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9249befe-1afe-43e4-97b4-cb9ee45faac2" containerName="revoke-keys" Nov 03 16:37:10.552212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.552169 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9249befe-1afe-43e4-97b4-cb9ee45faac2" containerName="revoke-keys" Nov 03 16:37:10.552264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.552255 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9249befe-1afe-43e4-97b4-cb9ee45faac2" containerName="revoke-keys" Nov 03 16:37:10.555581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.555565 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.571787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.571754 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:37:10.574830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.574797 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck"] Nov 03 16:37:10.578279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.578252 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.578397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.578306 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fspns\" (UniqueName: \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.578471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.578407 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.584585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.584557 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:37:10.584718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.584569 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:37:10.584718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.584643 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:37:10.679433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.679394 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.679650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.679474 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fspns\" (UniqueName: \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.679650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.679548 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.680213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.680186 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.681918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.681896 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.703536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.703502 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fspns\" (UniqueName: \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\") pod \"maas-api-revoke-sub-35f3c711-gr9ck\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:10.869781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:10.869680 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:11.180508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:11.180477 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck"] Nov 03 16:37:11.180779 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:11.180752 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea074ca_583b_4255_8c99_970d911753e8.slice/crio-46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82 WatchSource:0}: Error finding container 46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82: Status 404 returned error can't find the container with id 46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82 Nov 03 16:37:11.577089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:11.577053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" event={"ID":"dea074ca-583b-4255-8c99-970d911753e8","Type":"ContainerStarted","Data":"64d298fb7bd0c595a907aea76042e07ef5341635753d9e2c17b8c3fed96c6beb"} Nov 03 16:37:11.577542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:11.577106 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" event={"ID":"dea074ca-583b-4255-8c99-970d911753e8","Type":"ContainerStarted","Data":"46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82"} Nov 03 16:37:11.633191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:11.633131 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" podStartSLOduration=1.633116521 podStartE2EDuration="1.633116521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:11.619400075 +0000 UTC m=+1326.952613646" watchObservedRunningTime="2026-11-03 16:37:11.633116521 +0000 UTC m=+1326.966330090" Nov 03 16:37:12.585829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:12.585795 6001 generic.go:356] "Generic (PLEG): container finished" podID="dea074ca-583b-4255-8c99-970d911753e8" containerID="64d298fb7bd0c595a907aea76042e07ef5341635753d9e2c17b8c3fed96c6beb" exitCode=0 Nov 03 16:37:12.586289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:12.585892 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" event={"ID":"dea074ca-583b-4255-8c99-970d911753e8","Type":"ContainerDied","Data":"64d298fb7bd0c595a907aea76042e07ef5341635753d9e2c17b8c3fed96c6beb"} Nov 03 16:37:13.459765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.459728 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:13.463847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.463824 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.481860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.481831 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 16:37:13.481860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.481842 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 16:37:13.494390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.494360 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 16:37:13.494603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.494576 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-061ea8950c76\"" Nov 03 16:37:13.494725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.494656 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qdtlb\"" Nov 03 16:37:13.494808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.494757 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d4a61e3530c6\"" Nov 03 16:37:13.494808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.494783 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 16:37:13.497269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.497244 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:13.518517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.518490 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.518636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.518531 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghch\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.518636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.518633 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.518761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.518709 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.625372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.623889 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghch\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.625372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.624003 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.625372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.624057 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.625372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.624219 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.631087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.631059 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.631825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.631579 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.632009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.631642 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.654249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.654218 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghch\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\") pod \"payload-processing-external-model-79b6f456c4-8zv6j\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.777302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.777270 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:13.814414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.814382 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:13.934263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.934223 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\") pod \"dea074ca-583b-4255-8c99-970d911753e8\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " Nov 03 16:37:13.934433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.934300 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\" (UniqueName: \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\") pod \"dea074ca-583b-4255-8c99-970d911753e8\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " Nov 03 16:37:13.934433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.934393 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\") pod \"dea074ca-583b-4255-8c99-970d911753e8\" (UID: \"dea074ca-583b-4255-8c99-970d911753e8\") " Nov 03 16:37:13.934718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.934692 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca" pod "dea074ca-583b-4255-8c99-970d911753e8" (UID: "dea074ca-583b-4255-8c99-970d911753e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:13.934929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.934908 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea074ca-583b-4255-8c99-970d911753e8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:13.936511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.936492 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns" pod "dea074ca-583b-4255-8c99-970d911753e8" (UID: "dea074ca-583b-4255-8c99-970d911753e8"). InnerVolumeSpecName "kube-api-access-fspns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:13.936607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:13.936574 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca" pod "dea074ca-583b-4255-8c99-970d911753e8" (UID: "dea074ca-583b-4255-8c99-970d911753e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:14.036397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.036301 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fspns\" (UniqueName: \"kubernetes.io/projected/dea074ca-583b-4255-8c99-970d911753e8-kube-api-access-fspns\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:14.036397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.036333 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea074ca-583b-4255-8c99-970d911753e8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:14.059484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.059438 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:14.059986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.059974 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dea074ca-583b-4255-8c99-970d911753e8" containerName="revoke-keys" Nov 03 16:37:14.060041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.059988 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea074ca-583b-4255-8c99-970d911753e8" containerName="revoke-keys" Nov 03 16:37:14.060106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.060098 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea074ca-583b-4255-8c99-970d911753e8" containerName="revoke-keys" Nov 03 16:37:14.063606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.063587 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.083288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.083258 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5417ff19fed0\"" Nov 03 16:37:14.083480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.083346 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2bf65eb9e4e9\"" Nov 03 16:37:14.086823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.086797 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:14.090065 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:14.090034 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65a33f2_1bf7_4d13_bd02_842d414eff10.slice/crio-b4b278353a910eda9f7c60b15b5002549e16a55bdda5aa8c61e373fcd695329e WatchSource:0}: Error finding container b4b278353a910eda9f7c60b15b5002549e16a55bdda5aa8c61e373fcd695329e: Status 404 returned error can't find the container with id b4b278353a910eda9f7c60b15b5002549e16a55bdda5aa8c61e373fcd695329e Nov 03 16:37:14.090371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.090345 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:14.137314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.137285 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.137439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.137404 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.137530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.137437 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss4zs\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.137530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.137520 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.239080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.239040 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.239269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.239132 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.239269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.239197 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.239269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.239228 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss4zs\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.240043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.240015 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.240160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.240060 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.241823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.241797 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.264920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.264889 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss4zs\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\") pod \"payload-processing-external-model-7cf8d5c7d-tzzcf\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.378105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.378019 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:14.647152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.647079 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" Nov 03 16:37:14.647152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.647104 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck" event={"ID":"dea074ca-583b-4255-8c99-970d911753e8","Type":"ContainerDied","Data":"46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82"} Nov 03 16:37:14.647628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.647155 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46a91d2c625f97902d7da6e78643c73bbea1b96456b9f3940a2e46fc7dbb2c82" Nov 03 16:37:14.649034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.649012 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" event={"ID":"a65a33f2-1bf7-4d13-bd02-842d414eff10","Type":"ContainerStarted","Data":"1288789a547d182be2b0a00ea6e9ca2c5f6553a89c9d7e63c6826205596e3a1d"} Nov 03 16:37:14.649137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.649053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" event={"ID":"a65a33f2-1bf7-4d13-bd02-842d414eff10","Type":"ContainerStarted","Data":"b4b278353a910eda9f7c60b15b5002549e16a55bdda5aa8c61e373fcd695329e"} Nov 03 16:37:14.649238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.649209 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:14.695463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.695393 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" podStartSLOduration=1.695379901 podStartE2EDuration="1.695379901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:14.693509873 +0000 UTC m=+1330.026723443" watchObservedRunningTime="2026-11-03 16:37:14.695379901 +0000 UTC m=+1330.028593469" Nov 03 16:37:14.705755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:14.705733 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:14.706051 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:14.706020 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd60024f_a5a9_4573_b631_c80a57c3f1c8.slice/crio-550b98da9c4a3d82b83f90b925098dabeed64afdfb25c617e6b98770739573ef WatchSource:0}: Error finding container 550b98da9c4a3d82b83f90b925098dabeed64afdfb25c617e6b98770739573ef: Status 404 returned error can't find the container with id 550b98da9c4a3d82b83f90b925098dabeed64afdfb25c617e6b98770739573ef Nov 03 16:37:15.657392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:15.657352 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" event={"ID":"fd60024f-a5a9-4573-b631-c80a57c3f1c8","Type":"ContainerStarted","Data":"4155f050c43d35bcf0aefbcbc90c5d981578d2eb37364bf5434763a6fd3f9b2d"} Nov 03 16:37:15.657784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:15.657464 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" event={"ID":"fd60024f-a5a9-4573-b631-c80a57c3f1c8","Type":"ContainerStarted","Data":"550b98da9c4a3d82b83f90b925098dabeed64afdfb25c617e6b98770739573ef"} Nov 03 16:37:15.657784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:15.657649 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:15.703798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:15.703743 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" podStartSLOduration=1.703727225 podStartE2EDuration="1.703727225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:15.701963071 +0000 UTC m=+1331.035176639" watchObservedRunningTime="2026-11-03 16:37:15.703727225 +0000 UTC m=+1331.036940793" Nov 03 16:37:22.266466 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:37:22.266346 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:37:25.660937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:25.660906 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:26.667105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:26.667076 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:26.757981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:26.757943 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:31.760268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:31.759533 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" podUID="a65a33f2-1bf7-4d13-bd02-842d414eff10" containerName="payload-processing" containerID="cri-o://1288789a547d182be2b0a00ea6e9ca2c5f6553a89c9d7e63c6826205596e3a1d" gracePeriod=25 Nov 03 16:37:31.935663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:31.935617 6001 generic.go:356] "Generic (PLEG): container finished" podID="a65a33f2-1bf7-4d13-bd02-842d414eff10" containerID="1288789a547d182be2b0a00ea6e9ca2c5f6553a89c9d7e63c6826205596e3a1d" exitCode=0 Nov 03 16:37:31.935875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:31.935785 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" event={"ID":"a65a33f2-1bf7-4d13-bd02-842d414eff10","Type":"ContainerDied","Data":"1288789a547d182be2b0a00ea6e9ca2c5f6553a89c9d7e63c6826205596e3a1d"} Nov 03 16:37:31.991977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:31.991585 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:37:31.991977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:31.991653 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:37:32.122109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.122076 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:32.222817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.222781 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\") pod \"a65a33f2-1bf7-4d13-bd02-842d414eff10\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " Nov 03 16:37:32.222817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.222831 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\") pod \"a65a33f2-1bf7-4d13-bd02-842d414eff10\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " Nov 03 16:37:32.223102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.222863 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\") pod \"a65a33f2-1bf7-4d13-bd02-842d414eff10\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " Nov 03 16:37:32.223102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.222955 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\") pod \"a65a33f2-1bf7-4d13-bd02-842d414eff10\" (UID: \"a65a33f2-1bf7-4d13-bd02-842d414eff10\") " Nov 03 16:37:32.223546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.223277 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume" pod "a65a33f2-1bf7-4d13-bd02-842d414eff10" (UID: "a65a33f2-1bf7-4d13-bd02-842d414eff10"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:32.223717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.223686 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume" pod "a65a33f2-1bf7-4d13-bd02-842d414eff10" (UID: "a65a33f2-1bf7-4d13-bd02-842d414eff10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:32.223825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.223767 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:32.225890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.225865 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials" pod "a65a33f2-1bf7-4d13-bd02-842d414eff10" (UID: "a65a33f2-1bf7-4d13-bd02-842d414eff10"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:32.225997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.225899 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch" pod "a65a33f2-1bf7-4d13-bd02-842d414eff10" (UID: "a65a33f2-1bf7-4d13-bd02-842d414eff10"). InnerVolumeSpecName "kube-api-access-8ghch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:32.325736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.325651 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:32.325736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.325683 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a65a33f2-1bf7-4d13-bd02-842d414eff10-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:32.325736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.325694 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghch\" (UniqueName: \"kubernetes.io/projected/a65a33f2-1bf7-4d13-bd02-842d414eff10-kube-api-access-8ghch\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:32.944024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.943989 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" Nov 03 16:37:32.944455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.944023 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j" event={"ID":"a65a33f2-1bf7-4d13-bd02-842d414eff10","Type":"ContainerDied","Data":"b4b278353a910eda9f7c60b15b5002549e16a55bdda5aa8c61e373fcd695329e"} Nov 03 16:37:32.944455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.944090 6001 scope.go:122] "RemoveContainer" containerID="1288789a547d182be2b0a00ea6e9ca2c5f6553a89c9d7e63c6826205596e3a1d" Nov 03 16:37:32.998758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:32.998723 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:33.012026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.011993 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-79b6f456c4-8zv6j"] Nov 03 16:37:33.456242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.456209 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:37:33.456835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.456819 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a65a33f2-1bf7-4d13-bd02-842d414eff10" containerName="payload-processing" Nov 03 16:37:33.456900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.456836 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65a33f2-1bf7-4d13-bd02-842d414eff10" containerName="payload-processing" Nov 03 16:37:33.456939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.456921 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65a33f2-1bf7-4d13-bd02-842d414eff10" containerName="payload-processing" Nov 03 16:37:33.461619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.461602 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.480386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.480350 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:37:33.536974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.536938 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc8kz\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.536974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.536974 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.537192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.537098 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.537232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.537209 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.638100 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638052 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.638314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638166 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc8kz\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.638314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638200 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.638314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638294 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.638896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638866 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.639013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.638915 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.640967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.640942 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.662338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.662309 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc8kz\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\") pod \"payload-processing-external-model-59df74dc8-dss64\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:33.774289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:33.774257 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:34.091719 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:34.091693 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02548f4_0503_4944_8cc1_45bedd026771.slice/crio-022cd5e966192c0748ded03c63d34239261e07a2a2d2cadd041c7b336c0eb91f WatchSource:0}: Error finding container 022cd5e966192c0748ded03c63d34239261e07a2a2d2cadd041c7b336c0eb91f: Status 404 returned error can't find the container with id 022cd5e966192c0748ded03c63d34239261e07a2a2d2cadd041c7b336c0eb91f Nov 03 16:37:34.092771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:34.092742 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:37:34.790355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:34.790321 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a65a33f2-1bf7-4d13-bd02-842d414eff10" path="/var/lib/kubelet/pods/a65a33f2-1bf7-4d13-bd02-842d414eff10/volumes" Nov 03 16:37:34.959410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:34.959374 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" event={"ID":"b02548f4-0503-4944-8cc1-45bedd026771","Type":"ContainerStarted","Data":"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77"} Nov 03 16:37:34.959583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:34.959459 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" event={"ID":"b02548f4-0503-4944-8cc1-45bedd026771","Type":"ContainerStarted","Data":"022cd5e966192c0748ded03c63d34239261e07a2a2d2cadd041c7b336c0eb91f"} Nov 03 16:37:34.959583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:34.959524 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:35.009738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:35.009679 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" podStartSLOduration=2.009665248 podStartE2EDuration="2.009665248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:35.009418036 +0000 UTC m=+1350.342631605" watchObservedRunningTime="2026-11-03 16:37:35.009665248 +0000 UTC m=+1350.342878820" Nov 03 16:37:37.972254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.972206 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j"] Nov 03 16:37:37.977794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.977767 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:37.989158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.989124 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:37.989314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.989232 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:37.989401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.989385 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnkph\" (UniqueName: \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:37.996022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:37.995996 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j"] Nov 03 16:37:38.006461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.005599 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:37:38.006461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.005638 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:37:38.006760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.006558 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:37:38.006760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.006659 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:37:38.090110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.090072 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnkph\" (UniqueName: \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.090110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.090113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.090348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.090170 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.090963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.090935 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.092564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.092542 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.115075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.115041 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnkph\" (UniqueName: \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\") pod \"maas-api-revoke-sub-a056ea95-hxc5j\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.292156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.292060 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:38.592688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.592649 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j"] Nov 03 16:37:38.593162 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:38.593134 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0880448a_0f2a_411e_a56d_be8e21ac6539.slice/crio-c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550 WatchSource:0}: Error finding container c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550: Status 404 returned error can't find the container with id c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550 Nov 03 16:37:38.994279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.994231 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" event={"ID":"0880448a-0f2a-411e-a56d-be8e21ac6539","Type":"ContainerStarted","Data":"d613c4f35fdfad495628363ff9a9bf364aeff65f49826d831b8afbaece611aa1"} Nov 03 16:37:38.994952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:38.994306 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" event={"ID":"0880448a-0f2a-411e-a56d-be8e21ac6539","Type":"ContainerStarted","Data":"c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550"} Nov 03 16:37:39.039541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:39.039474 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" podStartSLOduration=2.039435912 podStartE2EDuration="2.039435912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:39.036578466 +0000 UTC m=+1354.369792036" watchObservedRunningTime="2026-11-03 16:37:39.039435912 +0000 UTC m=+1354.372649483" Nov 03 16:37:40.007289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:40.006802 6001 generic.go:356] "Generic (PLEG): container finished" podID="0880448a-0f2a-411e-a56d-be8e21ac6539" containerID="d613c4f35fdfad495628363ff9a9bf364aeff65f49826d831b8afbaece611aa1" exitCode=0 Nov 03 16:37:40.007289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:40.006878 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" event={"ID":"0880448a-0f2a-411e-a56d-be8e21ac6539","Type":"ContainerDied","Data":"d613c4f35fdfad495628363ff9a9bf364aeff65f49826d831b8afbaece611aa1"} Nov 03 16:37:41.833708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:41.833667 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:42.132469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.123925 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\") pod \"0880448a-0f2a-411e-a56d-be8e21ac6539\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " Nov 03 16:37:42.132894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.132857 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\") pod \"0880448a-0f2a-411e-a56d-be8e21ac6539\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " Nov 03 16:37:42.133164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.133144 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\" (UniqueName: \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\") pod \"0880448a-0f2a-411e-a56d-be8e21ac6539\" (UID: \"0880448a-0f2a-411e-a56d-be8e21ac6539\") " Nov 03 16:37:42.140223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.140186 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca" pod "0880448a-0f2a-411e-a56d-be8e21ac6539" (UID: "0880448a-0f2a-411e-a56d-be8e21ac6539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:42.154115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.154077 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph" pod "0880448a-0f2a-411e-a56d-be8e21ac6539" (UID: "0880448a-0f2a-411e-a56d-be8e21ac6539"). InnerVolumeSpecName "kube-api-access-qnkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:42.173033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.172978 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca" pod "0880448a-0f2a-411e-a56d-be8e21ac6539" (UID: "0880448a-0f2a-411e-a56d-be8e21ac6539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:37:42.210483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.201626 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" event={"ID":"0880448a-0f2a-411e-a56d-be8e21ac6539","Type":"ContainerDied","Data":"c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550"} Nov 03 16:37:42.210483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.201697 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11154d3ce24a4b61092bb07198bf1fd76e8ff988e65f98e69393e6a97014550" Nov 03 16:37:42.210483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.201773 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j" Nov 03 16:37:42.250995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.246515 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:42.251312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.251286 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0880448a-0f2a-411e-a56d-be8e21ac6539-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:42.255545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:42.255497 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnkph\" (UniqueName: \"kubernetes.io/projected/0880448a-0f2a-411e-a56d-be8e21ac6539-kube-api-access-qnkph\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:44.196250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.196209 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 16:37:44.197015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.196967 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0880448a-0f2a-411e-a56d-be8e21ac6539" containerName="revoke-keys" Nov 03 16:37:44.197015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.196991 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0880448a-0f2a-411e-a56d-be8e21ac6539" containerName="revoke-keys" Nov 03 16:37:44.197206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.197192 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0880448a-0f2a-411e-a56d-be8e21ac6539" containerName="revoke-keys" Nov 03 16:37:44.201193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.201175 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.221510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.221474 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 16:37:44.231390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.231360 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"distinct-w3-3a2fa2-kserve-self-signed-certs\"" Nov 03 16:37:44.298759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.298720 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.298975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.298846 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.298975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.298919 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.298975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.298950 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdsq7\" (UniqueName: \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.299084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.299035 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.299084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.299056 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400055 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400181 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400291 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400334 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdsq7\" (UniqueName: \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400395 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.400673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.400423 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.401843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.401789 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.401843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.401802 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.404292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.402070 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.406196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.406165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.415278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.415217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.428511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.428424 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdsq7\" (UniqueName: \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\") pod \"distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.517053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.517011 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:44.842751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:44.842727 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 16:37:44.843920 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:44.843897 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd24aba52_e346_44d7_a9b7_1d523e330b71.slice/crio-4303d9156a864f6265df679fa4b4628ea6ceeb888e685f4ea708cc2e99ca298c WatchSource:0}: Error finding container 4303d9156a864f6265df679fa4b4628ea6ceeb888e685f4ea708cc2e99ca298c: Status 404 returned error can't find the container with id 4303d9156a864f6265df679fa4b4628ea6ceeb888e685f4ea708cc2e99ca298c Nov 03 16:37:45.087701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.087668 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:37:45.091802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.091744 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110088 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110113 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110244 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110278 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110303 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110460 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110597 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110573 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf4fb\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.110699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.110676 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.135488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.128706 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3a2fa2-openshift-default-dockercfg-q9n76\"" Nov 03 16:37:45.136698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.136663 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:37:45.212069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212039 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212087 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212107 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212132 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212248 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf4fb\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212292 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212420 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212490 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212499 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212733 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212804 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.212928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.212871 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.213027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.213014 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.214723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.214705 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.214850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.214832 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.226929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.226900 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" event={"ID":"d24aba52-e346-44d7-a9b7-1d523e330b71","Type":"ContainerStarted","Data":"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509"} Nov 03 16:37:45.227016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.226946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" event={"ID":"d24aba52-e346-44d7-a9b7-1d523e330b71","Type":"ContainerStarted","Data":"4303d9156a864f6265df679fa4b4628ea6ceeb888e685f4ea708cc2e99ca298c"} Nov 03 16:37:45.236354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.236329 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.237525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.237505 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf4fb\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.271856 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.271810 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podStartSLOduration=1.271798874 podStartE2EDuration="1.271798874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:45.270229773 +0000 UTC m=+1360.603443342" watchObservedRunningTime="2026-11-03 16:37:45.271798874 +0000 UTC m=+1360.605012484" Nov 03 16:37:45.408583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.408489 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:45.517669 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.517627 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:45.732538 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:37:45.732508 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1795765b_ab43_4308_9277_636688cd2380.slice/crio-ca2cd34775a6e7f4ca99d867d52df625c46f87a537f7eaa92d80cba02140baf8 WatchSource:0}: Error finding container ca2cd34775a6e7f4ca99d867d52df625c46f87a537f7eaa92d80cba02140baf8: Status 404 returned error can't find the container with id ca2cd34775a6e7f4ca99d867d52df625c46f87a537f7eaa92d80cba02140baf8 Nov 03 16:37:45.732538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.732518 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:37:45.735359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.735328 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:45.735454 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.735390 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:45.735454 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.735419 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:37:45.970387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:45.970347 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:37:46.063429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.055942 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:46.238884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.238839 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" event={"ID":"1795765b-ab43-4308-9277-636688cd2380","Type":"ContainerStarted","Data":"5e68d43c9361dcbccb212ef99334933ead2ba1e5beb276a3698acdcd1b5b7cc5"} Nov 03 16:37:46.239325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.238914 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" event={"ID":"1795765b-ab43-4308-9277-636688cd2380","Type":"ContainerStarted","Data":"ca2cd34775a6e7f4ca99d867d52df625c46f87a537f7eaa92d80cba02140baf8"} Nov 03 16:37:46.289191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.289138 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podStartSLOduration=1.289122316 podStartE2EDuration="1.289122316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:46.286877029 +0000 UTC m=+1361.620090611" watchObservedRunningTime="2026-11-03 16:37:46.289122316 +0000 UTC m=+1361.622335884" Nov 03 16:37:46.409337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.409255 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:46.411097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.411065 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:46.411231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:46.411114 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:47.409322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:47.409283 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:47.409741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:47.409338 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:48.409184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:48.409140 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:48.409367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:48.409204 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:49.409429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:49.409391 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:49.409887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:49.409483 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:50.409725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:50.409687 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:50.410137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:50.409746 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:51.060553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.060508 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" podUID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" containerName="payload-processing" containerID="cri-o://4155f050c43d35bcf0aefbcbc90c5d981578d2eb37364bf5434763a6fd3f9b2d" gracePeriod=25 Nov 03 16:37:51.296984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.296660 6001 generic.go:356] "Generic (PLEG): container finished" podID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" containerID="4155f050c43d35bcf0aefbcbc90c5d981578d2eb37364bf5434763a6fd3f9b2d" exitCode=0 Nov 03 16:37:51.296984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.296883 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" event={"ID":"fd60024f-a5a9-4573-b631-c80a57c3f1c8","Type":"ContainerDied","Data":"4155f050c43d35bcf0aefbcbc90c5d981578d2eb37364bf5434763a6fd3f9b2d"} Nov 03 16:37:51.374521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.374488 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:51.409682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.409646 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:51.409854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.409707 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:51.518849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.518809 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\") pod \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " Nov 03 16:37:51.519051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.518912 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\") pod \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " Nov 03 16:37:51.519051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.518945 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\") pod \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " Nov 03 16:37:51.519051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.518994 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\") pod \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\" (UID: \"fd60024f-a5a9-4573-b631-c80a57c3f1c8\") " Nov 03 16:37:51.519474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.519412 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume" pod "fd60024f-a5a9-4573-b631-c80a57c3f1c8" (UID: "fd60024f-a5a9-4573-b631-c80a57c3f1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:51.519604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.519418 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume" pod "fd60024f-a5a9-4573-b631-c80a57c3f1c8" (UID: "fd60024f-a5a9-4573-b631-c80a57c3f1c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:37:51.521226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.521196 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs" pod "fd60024f-a5a9-4573-b631-c80a57c3f1c8" (UID: "fd60024f-a5a9-4573-b631-c80a57c3f1c8"). InnerVolumeSpecName "kube-api-access-ss4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:51.521321 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.521224 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials" pod "fd60024f-a5a9-4573-b631-c80a57c3f1c8" (UID: "fd60024f-a5a9-4573-b631-c80a57c3f1c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:37:51.620940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.620834 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:51.620940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.620892 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:51.620940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.620905 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss4zs\" (UniqueName: \"kubernetes.io/projected/fd60024f-a5a9-4573-b631-c80a57c3f1c8-kube-api-access-ss4zs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:51.620940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:51.620913 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fd60024f-a5a9-4573-b631-c80a57c3f1c8-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:37:52.250476 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:37:52.250412 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:37:52.306792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.306758 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" event={"ID":"fd60024f-a5a9-4573-b631-c80a57c3f1c8","Type":"ContainerDied","Data":"550b98da9c4a3d82b83f90b925098dabeed64afdfb25c617e6b98770739573ef"} Nov 03 16:37:52.306792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.306786 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf" Nov 03 16:37:52.307014 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.306819 6001 scope.go:122] "RemoveContainer" containerID="4155f050c43d35bcf0aefbcbc90c5d981578d2eb37364bf5434763a6fd3f9b2d" Nov 03 16:37:52.361346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.361315 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:52.374064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.374036 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cf8d5c7d-tzzcf"] Nov 03 16:37:52.409823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.409796 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Nov 03 16:37:52.409922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.409852 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Nov 03 16:37:52.790554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:52.790518 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" path="/var/lib/kubelet/pods/fd60024f-a5a9-4573-b631-c80a57c3f1c8/volumes" Nov 03 16:37:53.413384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:53.413358 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:53.413902 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:53.413657 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:53.414525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:53.414507 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:37:54.518127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:54.517729 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:54.539030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:54.538976 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:55.375573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:55.375529 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:37:56.187929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.187886 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:37:56.189475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.189413 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" containerName="payload-processing" Nov 03 16:37:56.189679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.189659 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" containerName="payload-processing" Nov 03 16:37:56.189951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.189935 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd60024f-a5a9-4573-b631-c80a57c3f1c8" containerName="payload-processing" Nov 03 16:37:56.194354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.194329 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.213483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.212057 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d4a61e3530c6\"" Nov 03 16:37:56.217594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.217561 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:37:56.302574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.301956 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxws8\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.302574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.302018 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.302574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.302082 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.302574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.302221 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.414224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.414177 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.415943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.415751 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.416284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.416245 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxws8\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.416483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.416438 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.416685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.416651 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.417540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.417485 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.443697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.443623 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.464614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.464583 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxws8\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\") pod \"payload-processing-external-model-786c458675-mh55r\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:56.535469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:56.534729 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:57.228846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:57.227732 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:37:57.543963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:57.543736 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" event={"ID":"5690c64c-f0be-4cdf-8398-04babe85be99","Type":"ContainerStarted","Data":"98d073553a1e4fa95624833748fc404c859f6b8ced6ff77e57ba47f2d0288d1b"} Nov 03 16:37:58.560594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:58.558804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" event={"ID":"5690c64c-f0be-4cdf-8398-04babe85be99","Type":"ContainerStarted","Data":"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918"} Nov 03 16:37:58.560594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:58.559842 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:37:58.606306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:37:58.606226 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" podStartSLOduration=2.606204943 podStartE2EDuration="2.606204943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:58.603813098 +0000 UTC m=+1373.937026679" watchObservedRunningTime="2026-11-03 16:37:58.606204943 +0000 UTC m=+1373.939418513" Nov 03 16:38:01.083345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.083307 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:38:01.087655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.087632 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.121857 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.121796 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:38:01.145126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145086 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145214 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145314 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145357 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145490 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145522 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145636 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145679 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.145769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.145752 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw6fh\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246423 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246515 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246544 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246622 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246661 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw6fh\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246775 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.246899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246886 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.246956 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.247041 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.247171 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.247313 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.247372 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.247393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.247380 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.248892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.248871 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.249100 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.249082 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.270769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.270730 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.270973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.270947 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw6fh\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.402590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.402494 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:01.725548 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:01.725513 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8d5dda_fb6f_4913_9f87_8410d9ac54b3.slice/crio-c1601b602fc758236ce74dd20892d42a158fa355e509964fda5764b95e87a00e WatchSource:0}: Error finding container c1601b602fc758236ce74dd20892d42a158fa355e509964fda5764b95e87a00e: Status 404 returned error can't find the container with id c1601b602fc758236ce74dd20892d42a158fa355e509964fda5764b95e87a00e Nov 03 16:38:01.726106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.726078 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:38:01.728311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.728282 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:01.728396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.728348 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:01.728396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:01.728380 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:02.587802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:02.587758 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" event={"ID":"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3","Type":"ContainerStarted","Data":"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0"} Nov 03 16:38:02.588287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:02.587821 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" event={"ID":"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3","Type":"ContainerStarted","Data":"c1601b602fc758236ce74dd20892d42a158fa355e509964fda5764b95e87a00e"} Nov 03 16:38:02.671371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:02.671298 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podStartSLOduration=1.671278085 podStartE2EDuration="1.671278085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:02.670217194 +0000 UTC m=+1378.003430775" watchObservedRunningTime="2026-11-03 16:38:02.671278085 +0000 UTC m=+1378.004491655" Nov 03 16:38:03.403581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:03.403550 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:03.405401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:03.405373 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:03.405553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:03.405426 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:04.403798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:04.403753 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:04.404209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:04.403840 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:05.408631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:05.408528 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:05.409060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:05.408617 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:06.403212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:06.403176 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:06.403392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:06.403230 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:07.403611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:07.403570 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:07.404113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:07.403624 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:08.403722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:08.403665 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Nov 03 16:38:08.404402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:08.404367 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Nov 03 16:38:09.512812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:09.512779 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:09.514851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:09.514829 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:09.521424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:09.521393 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:38:09.594665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:09.594633 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:38:10.250468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:10.178751 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:38:11.109221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:11.108087 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:38:11.182600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:11.182538 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:11.208905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:11.208850 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 16:38:11.406807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:11.406662 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:12.062650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:12.062605 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:12.185016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:12.184974 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:38:12.205468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:12.186126 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:38:12.241387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:12.241358 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:38:15.180233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.180168 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" podUID="b02548f4-0503-4944-8cc1-45bedd026771" containerName="payload-processing" containerID="cri-o://7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77" gracePeriod=25 Nov 03 16:38:15.455028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.454923 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:38:15.549567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.549525 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\") pod \"b02548f4-0503-4944-8cc1-45bedd026771\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " Nov 03 16:38:15.549744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.549580 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\") pod \"b02548f4-0503-4944-8cc1-45bedd026771\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " Nov 03 16:38:15.549744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.549693 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\") pod \"b02548f4-0503-4944-8cc1-45bedd026771\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " Nov 03 16:38:15.549744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.549731 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\") pod \"b02548f4-0503-4944-8cc1-45bedd026771\" (UID: \"b02548f4-0503-4944-8cc1-45bedd026771\") " Nov 03 16:38:15.550105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.549978 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume" pod "b02548f4-0503-4944-8cc1-45bedd026771" (UID: "b02548f4-0503-4944-8cc1-45bedd026771"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:15.550237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.550174 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume" pod "b02548f4-0503-4944-8cc1-45bedd026771" (UID: "b02548f4-0503-4944-8cc1-45bedd026771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:15.550311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.550271 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:15.550311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.550284 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b02548f4-0503-4944-8cc1-45bedd026771-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:15.551873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.551848 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials" pod "b02548f4-0503-4944-8cc1-45bedd026771" (UID: "b02548f4-0503-4944-8cc1-45bedd026771"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:15.551973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.551876 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz" pod "b02548f4-0503-4944-8cc1-45bedd026771" (UID: "b02548f4-0503-4944-8cc1-45bedd026771"). InnerVolumeSpecName "kube-api-access-nc8kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:15.651809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.651771 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:15.651809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.651805 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc8kz\" (UniqueName: \"kubernetes.io/projected/b02548f4-0503-4944-8cc1-45bedd026771-kube-api-access-nc8kz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:15.917915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.917865 6001 generic.go:356] "Generic (PLEG): container finished" podID="b02548f4-0503-4944-8cc1-45bedd026771" containerID="7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77" exitCode=0 Nov 03 16:38:15.917915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.917917 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" Nov 03 16:38:15.918184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.917920 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" event={"ID":"b02548f4-0503-4944-8cc1-45bedd026771","Type":"ContainerDied","Data":"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77"} Nov 03 16:38:15.918184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.917980 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-59df74dc8-dss64" event={"ID":"b02548f4-0503-4944-8cc1-45bedd026771","Type":"ContainerDied","Data":"022cd5e966192c0748ded03c63d34239261e07a2a2d2cadd041c7b336c0eb91f"} Nov 03 16:38:15.918184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.918005 6001 scope.go:122] "RemoveContainer" containerID="7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77" Nov 03 16:38:15.930415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.930388 6001 scope.go:122] "RemoveContainer" containerID="7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77" Nov 03 16:38:15.930844 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:38:15.930824 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77\": container with ID starting with 7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77 not found: ID does not exist" containerID="7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77" Nov 03 16:38:15.930947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.930852 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77"} err="failed to get container status \"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77\": rpc error: code = NotFound desc = could not find container \"7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77\": container with ID starting with 7f9793f53c773533d74d2ec139a3a4a76fd48d1a6b4f4eb64c9b2d3bd1b89d77 not found: ID does not exist" Nov 03 16:38:15.975765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.975726 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:38:15.986588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:15.986545 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-59df74dc8-dss64"] Nov 03 16:38:16.790426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:16.790387 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02548f4-0503-4944-8cc1-45bedd026771" path="/var/lib/kubelet/pods/b02548f4-0503-4944-8cc1-45bedd026771/volumes" Nov 03 16:38:17.611783 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.611731 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 16:38:17.612640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.612616 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b02548f4-0503-4944-8cc1-45bedd026771" containerName="payload-processing" Nov 03 16:38:17.612640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.612639 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02548f4-0503-4944-8cc1-45bedd026771" containerName="payload-processing" Nov 03 16:38:17.612895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.612882 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02548f4-0503-4944-8cc1-45bedd026771" containerName="payload-processing" Nov 03 16:38:17.618595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.618561 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.633709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.633681 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 16:38:17.641923 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.641891 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"distinct-2-w3-3a2fa2-kserve-self-signed-certs\"" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.687770 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxm6\" (UniqueName: \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.687920 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.688021 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.688130 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.688225 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.688466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.688380 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791217 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxm6\" (UniqueName: \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791283 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791319 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791363 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791408 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791495 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.791990 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.792186 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.792802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.792398 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.808267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.808186 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.823472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.822897 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.827500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.825905 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxm6\" (UniqueName: \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\") pod \"distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:17.939377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:17.938726 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:18.248839 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:18.248805 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9afaa6bd_8c35_426f_9ba9_227a57feaf3f.slice/crio-4b59ff35ec8f18ce1f7ea83cb3c0e444be83889b5d3ccf0bfe94f78e7430e5a5 WatchSource:0}: Error finding container 4b59ff35ec8f18ce1f7ea83cb3c0e444be83889b5d3ccf0bfe94f78e7430e5a5: Status 404 returned error can't find the container with id 4b59ff35ec8f18ce1f7ea83cb3c0e444be83889b5d3ccf0bfe94f78e7430e5a5 Nov 03 16:38:18.249093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.249068 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 16:38:18.550778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.550687 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5"] Nov 03 16:38:18.560222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.556526 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.573704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.573674 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:38:18.573887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.573757 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:38:18.576402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.576373 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5"] Nov 03 16:38:18.586050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.586018 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:38:18.586182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.586079 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:38:18.601777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.601740 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.601940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.601891 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.602035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.602013 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvlb\" (UniqueName: \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.703416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.703381 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.703619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.703495 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.703619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.703539 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvlb\" (UniqueName: \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.704191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.704165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.705790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.705763 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.727963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.727939 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvlb\" (UniqueName: \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\") pod \"maas-api-revoke-sub-33fe5226-vp6r5\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.872298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.872212 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:18.953508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.953473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" event={"ID":"9afaa6bd-8c35-426f-9ba9-227a57feaf3f","Type":"ContainerStarted","Data":"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01"} Nov 03 16:38:18.953690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.953527 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" event={"ID":"9afaa6bd-8c35-426f-9ba9-227a57feaf3f","Type":"ContainerStarted","Data":"4b59ff35ec8f18ce1f7ea83cb3c0e444be83889b5d3ccf0bfe94f78e7430e5a5"} Nov 03 16:38:18.997842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:18.997793 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podStartSLOduration=1.997778437 podStartE2EDuration="1.997778437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:18.997231273 +0000 UTC m=+1394.330444842" watchObservedRunningTime="2026-11-03 16:38:18.997778437 +0000 UTC m=+1394.330992006" Nov 03 16:38:19.172900 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:19.172857 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a87945_c57f_4250_bd3b_f13493141b24.slice/crio-fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929 WatchSource:0}: Error finding container fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929: Status 404 returned error can't find the container with id fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929 Nov 03 16:38:19.173927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:19.173786 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5"] Nov 03 16:38:19.939833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:19.939796 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:19.961753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:19.961716 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" event={"ID":"54a87945-c57f-4250-bd3b-f13493141b24","Type":"ContainerStarted","Data":"b7b8e109e93ffa436f22cc3d35fe91b9adb4aa5c4cec83fae72d5eecd60ba0fe"} Nov 03 16:38:19.961934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:19.961780 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" event={"ID":"54a87945-c57f-4250-bd3b-f13493141b24","Type":"ContainerStarted","Data":"fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929"} Nov 03 16:38:20.006331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:20.006283 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" podStartSLOduration=2.006270359 podStartE2EDuration="2.006270359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:20.003938921 +0000 UTC m=+1395.337152500" watchObservedRunningTime="2026-11-03 16:38:20.006270359 +0000 UTC m=+1395.339483927" Nov 03 16:38:20.969976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:20.969940 6001 generic.go:356] "Generic (PLEG): container finished" podID="54a87945-c57f-4250-bd3b-f13493141b24" containerID="b7b8e109e93ffa436f22cc3d35fe91b9adb4aa5c4cec83fae72d5eecd60ba0fe" exitCode=0 Nov 03 16:38:20.970381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:20.970021 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" event={"ID":"54a87945-c57f-4250-bd3b-f13493141b24","Type":"ContainerDied","Data":"b7b8e109e93ffa436f22cc3d35fe91b9adb4aa5c4cec83fae72d5eecd60ba0fe"} Nov 03 16:38:22.130858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.130826 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:22.246392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.246294 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\" (UniqueName: \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\") pod \"54a87945-c57f-4250-bd3b-f13493141b24\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " Nov 03 16:38:22.246570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.246440 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\") pod \"54a87945-c57f-4250-bd3b-f13493141b24\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " Nov 03 16:38:22.246570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.246527 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\") pod \"54a87945-c57f-4250-bd3b-f13493141b24\" (UID: \"54a87945-c57f-4250-bd3b-f13493141b24\") " Nov 03 16:38:22.247192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.246822 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca" pod "54a87945-c57f-4250-bd3b-f13493141b24" (UID: "54a87945-c57f-4250-bd3b-f13493141b24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:22.247192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.247157 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:22.249035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.249005 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca" pod "54a87945-c57f-4250-bd3b-f13493141b24" (UID: "54a87945-c57f-4250-bd3b-f13493141b24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:22.249150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.249039 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb" pod "54a87945-c57f-4250-bd3b-f13493141b24" (UID: "54a87945-c57f-4250-bd3b-f13493141b24"). InnerVolumeSpecName "kube-api-access-2rvlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:22.250833 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:38:22.250798 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:38:22.348409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.348374 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvlb\" (UniqueName: \"kubernetes.io/projected/54a87945-c57f-4250-bd3b-f13493141b24-kube-api-access-2rvlb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:22.348409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.348404 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54a87945-c57f-4250-bd3b-f13493141b24-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:22.990526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.990494 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" Nov 03 16:38:22.990526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.990511 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5" event={"ID":"54a87945-c57f-4250-bd3b-f13493141b24","Type":"ContainerDied","Data":"fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929"} Nov 03 16:38:22.990815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:22.990558 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbd8b030f04c2dafbcb8a2ab6b5ca27fc7e5728e686c3eeb77964362603e7929" Nov 03 16:38:27.940305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:27.940267 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:27.953182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:27.953154 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:28.039150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:28.039119 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:38:30.139376 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:30.139343 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:38:35.140545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.140493 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" podUID="5690c64c-f0be-4cdf-8398-04babe85be99" containerName="payload-processing" containerID="cri-o://41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918" gracePeriod=25 Nov 03 16:38:35.419651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.419625 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:38:35.516230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516189 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\") pod \"5690c64c-f0be-4cdf-8398-04babe85be99\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " Nov 03 16:38:35.516413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516389 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\") pod \"5690c64c-f0be-4cdf-8398-04babe85be99\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " Nov 03 16:38:35.516480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516431 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\") pod \"5690c64c-f0be-4cdf-8398-04babe85be99\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " Nov 03 16:38:35.516549 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516530 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume" pod "5690c64c-f0be-4cdf-8398-04babe85be99" (UID: "5690c64c-f0be-4cdf-8398-04babe85be99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:35.516591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516557 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\") pod \"5690c64c-f0be-4cdf-8398-04babe85be99\" (UID: \"5690c64c-f0be-4cdf-8398-04babe85be99\") " Nov 03 16:38:35.516880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516855 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume" pod "5690c64c-f0be-4cdf-8398-04babe85be99" (UID: "5690c64c-f0be-4cdf-8398-04babe85be99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:35.516981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516966 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:35.517033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.516983 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5690c64c-f0be-4cdf-8398-04babe85be99-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:35.518672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.518653 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8" pod "5690c64c-f0be-4cdf-8398-04babe85be99" (UID: "5690c64c-f0be-4cdf-8398-04babe85be99"). InnerVolumeSpecName "kube-api-access-dxws8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:35.518782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.518767 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials" pod "5690c64c-f0be-4cdf-8398-04babe85be99" (UID: "5690c64c-f0be-4cdf-8398-04babe85be99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:35.618438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.618403 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:35.618438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:35.618434 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxws8\" (UniqueName: \"kubernetes.io/projected/5690c64c-f0be-4cdf-8398-04babe85be99-kube-api-access-dxws8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:36.085466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.084322 6001 generic.go:356] "Generic (PLEG): container finished" podID="5690c64c-f0be-4cdf-8398-04babe85be99" containerID="41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918" exitCode=0 Nov 03 16:38:36.085466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.085374 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" event={"ID":"5690c64c-f0be-4cdf-8398-04babe85be99","Type":"ContainerDied","Data":"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918"} Nov 03 16:38:36.085466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.085435 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" event={"ID":"5690c64c-f0be-4cdf-8398-04babe85be99","Type":"ContainerDied","Data":"98d073553a1e4fa95624833748fc404c859f6b8ced6ff77e57ba47f2d0288d1b"} Nov 03 16:38:36.085791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.085484 6001 scope.go:122] "RemoveContainer" containerID="41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918" Nov 03 16:38:36.085791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.085661 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-786c458675-mh55r" Nov 03 16:38:36.176646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.176595 6001 scope.go:122] "RemoveContainer" containerID="41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918" Nov 03 16:38:36.177106 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:38:36.177075 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918\": container with ID starting with 41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918 not found: ID does not exist" containerID="41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918" Nov 03 16:38:36.177172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.177127 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918"} err="failed to get container status \"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918\": rpc error: code = NotFound desc = could not find container \"41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918\": container with ID starting with 41237b38747b01fa3ede61aa75b650a52c7dec2afad84a8a445a65672a53a918 not found: ID does not exist" Nov 03 16:38:36.184912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.184869 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:38:36.197258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:36.197219 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-786c458675-mh55r"] Nov 03 16:38:37.012470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:37.012248 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5690c64c-f0be-4cdf-8398-04babe85be99" path="/var/lib/kubelet/pods/5690c64c-f0be-4cdf-8398-04babe85be99/volumes" Nov 03 16:38:38.158470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:38.157665 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:45.296262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.296223 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 16:38:45.297064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297041 6001 cpu_manager.go:414] "containerMap: removing container" podUID="54a87945-c57f-4250-bd3b-f13493141b24" containerName="revoke-keys" Nov 03 16:38:45.297173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297066 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54a87945-c57f-4250-bd3b-f13493141b24" containerName="revoke-keys" Nov 03 16:38:45.297173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297116 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5690c64c-f0be-4cdf-8398-04babe85be99" containerName="payload-processing" Nov 03 16:38:45.297173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297125 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5690c64c-f0be-4cdf-8398-04babe85be99" containerName="payload-processing" Nov 03 16:38:45.297323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297279 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5690c64c-f0be-4cdf-8398-04babe85be99" containerName="payload-processing" Nov 03 16:38:45.297323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.297294 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="54a87945-c57f-4250-bd3b-f13493141b24" containerName="revoke-keys" Nov 03 16:38:45.310331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.310302 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.327787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.327754 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 16:38:45.429360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429320 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429375 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429398 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429529 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429563 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2km\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429732 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429621 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429750 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429772 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.429835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.429807 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.530928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.530888 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.530928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.530931 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.530967 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.530994 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531070 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531142 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2km\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531220 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531304 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531406 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531625 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.531883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.531858 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.533489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.533466 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.533595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.533531 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.555856 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.555784 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.555856 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.555786 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2km\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.645146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.645109 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:45.987715 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:45.987684 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6d9e26_425a_4478_9b90_6e5a9098626c.slice/crio-2e7975464a1c0961684577c4f91df59c0565a95d525e8b1cecb03fcfbc44355e WatchSource:0}: Error finding container 2e7975464a1c0961684577c4f91df59c0565a95d525e8b1cecb03fcfbc44355e: Status 404 returned error can't find the container with id 2e7975464a1c0961684577c4f91df59c0565a95d525e8b1cecb03fcfbc44355e Nov 03 16:38:45.987899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.987874 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 16:38:45.990431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.990398 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:45.990586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.990494 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:45.990586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:45.990542 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:38:46.433716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:46.433608 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" event={"ID":"4b6d9e26-425a-4478-9b90-6e5a9098626c","Type":"ContainerStarted","Data":"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418"} Nov 03 16:38:46.433716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:46.433665 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" event={"ID":"4b6d9e26-425a-4478-9b90-6e5a9098626c","Type":"ContainerStarted","Data":"2e7975464a1c0961684577c4f91df59c0565a95d525e8b1cecb03fcfbc44355e"} Nov 03 16:38:46.498517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:46.498436 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podStartSLOduration=1.498421142 podStartE2EDuration="1.498421142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:46.497091227 +0000 UTC m=+1421.830304807" watchObservedRunningTime="2026-11-03 16:38:46.498421142 +0000 UTC m=+1421.831634711" Nov 03 16:38:46.648587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:46.648541 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:47.624853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.624815 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp"] Nov 03 16:38:47.629194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.629174 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.668495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.665852 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:38:47.668495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.666027 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:38:47.668495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.666152 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:38:47.668495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.666223 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:38:47.669074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.669044 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:47.669293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.669265 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:47.677420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.677387 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp"] Nov 03 16:38:47.779256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.779215 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.779469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.779280 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbmqf\" (UniqueName: \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.779562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.779485 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.880765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.880679 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbmqf\" (UniqueName: \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.880999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.880975 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.881126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.881114 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.881634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.881611 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.883481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.883455 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.904673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.904644 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbmqf\" (UniqueName: \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\") pod \"maas-api-revoke-sub-3dd8bad3-z9kmp\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:47.956474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:47.953662 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:48.273428 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:48.273399 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8495e090_6715_4250_8022_11f0e88858c4.slice/crio-e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4 WatchSource:0}: Error finding container e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4: Status 404 returned error can't find the container with id e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4 Nov 03 16:38:48.274092 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:48.274066 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp"] Nov 03 16:38:48.484848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:48.484809 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" event={"ID":"8495e090-6715-4250-8022-11f0e88858c4","Type":"ContainerStarted","Data":"e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4"} Nov 03 16:38:48.650188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:48.648851 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:48.650188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:48.648918 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:49.493322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.493285 6001 generic.go:356] "Generic (PLEG): container finished" podID="8495e090-6715-4250-8022-11f0e88858c4" containerID="ec8d60cba2afa0547b04c031839ed8fdf55579f0888cf9352007dea56c47c44a" exitCode=0 Nov 03 16:38:49.493541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.493362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" event={"ID":"8495e090-6715-4250-8022-11f0e88858c4","Type":"ContainerDied","Data":"ec8d60cba2afa0547b04c031839ed8fdf55579f0888cf9352007dea56c47c44a"} Nov 03 16:38:49.656558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.649852 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:49.656558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.649918 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:49.872961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.872919 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 16:38:49.879175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.879142 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.883240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.883218 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 16:38:49.896181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.896160 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"openshift-service-ca.crt\"" Nov 03 16:38:49.896325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.896183 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"distinct-w0-71d7b4-kserve-self-signed-certs\"" Nov 03 16:38:49.896398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.896238 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"default-dockercfg-h4hqv\"" Nov 03 16:38:49.896494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.896255 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"kube-root-ca.crt\"" Nov 03 16:38:49.907111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907023 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjs2\" (UniqueName: \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.907250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907151 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.907250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907222 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.907374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907354 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.907434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907392 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:49.907554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:49.907437 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.008642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.008594 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.008642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.008649 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.008904 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.008698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.008904 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.008870 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjs2\" (UniqueName: \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.009012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.008936 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.009079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.009015 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.009137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.009108 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.009316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.009280 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.009425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.009396 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.011000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.010980 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.011174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.011157 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.034397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.034370 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjs2\" (UniqueName: \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\") pod \"distinct-w0-71d7b4-kserve-857c56657d-9h9zt\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.197891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.197803 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:50.530657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.530608 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 16:38:50.530818 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:38:50.530626 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59800126_e27d_4638_879e_77c88cc5a506.slice/crio-1c749f9e4960bc238782c12c084354e89f35303747893780c43ab98309955948 WatchSource:0}: Error finding container 1c749f9e4960bc238782c12c084354e89f35303747893780c43ab98309955948: Status 404 returned error can't find the container with id 1c749f9e4960bc238782c12c084354e89f35303747893780c43ab98309955948 Nov 03 16:38:50.665468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.660181 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:50.665468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.660242 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:50.764531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.747617 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:50.822105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.822067 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\") pod \"8495e090-6715-4250-8022-11f0e88858c4\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " Nov 03 16:38:50.822263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.822125 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\") pod \"8495e090-6715-4250-8022-11f0e88858c4\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " Nov 03 16:38:50.822302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.822262 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\" (UniqueName: \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\") pod \"8495e090-6715-4250-8022-11f0e88858c4\" (UID: \"8495e090-6715-4250-8022-11f0e88858c4\") " Nov 03 16:38:50.823433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.823399 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca" pod "8495e090-6715-4250-8022-11f0e88858c4" (UID: "8495e090-6715-4250-8022-11f0e88858c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:50.826012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.825988 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca" pod "8495e090-6715-4250-8022-11f0e88858c4" (UID: "8495e090-6715-4250-8022-11f0e88858c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:50.826107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.826027 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf" pod "8495e090-6715-4250-8022-11f0e88858c4" (UID: "8495e090-6715-4250-8022-11f0e88858c4"). InnerVolumeSpecName "kube-api-access-lbmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:50.924170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.924107 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8495e090-6715-4250-8022-11f0e88858c4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:50.924170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.924137 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8495e090-6715-4250-8022-11f0e88858c4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:50.924170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:50.924150 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbmqf\" (UniqueName: \"kubernetes.io/projected/8495e090-6715-4250-8022-11f0e88858c4-kube-api-access-lbmqf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:38:51.515356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.511426 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" Nov 03 16:38:51.515356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.511523 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp" event={"ID":"8495e090-6715-4250-8022-11f0e88858c4","Type":"ContainerDied","Data":"e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4"} Nov 03 16:38:51.515356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.511569 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2090c18e12a587f1eed00a57820df3bf4012265dd63ba4a6e3c23f90507d3e4" Nov 03 16:38:51.515356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.514129 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" event={"ID":"59800126-e27d-4638-879e-77c88cc5a506","Type":"ContainerStarted","Data":"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11"} Nov 03 16:38:51.515356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.514161 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" event={"ID":"59800126-e27d-4638-879e-77c88cc5a506","Type":"ContainerStarted","Data":"1c749f9e4960bc238782c12c084354e89f35303747893780c43ab98309955948"} Nov 03 16:38:51.571468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.564173 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podStartSLOduration=2.564151852 podStartE2EDuration="2.564151852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:51.560903354 +0000 UTC m=+1426.894116952" watchObservedRunningTime="2026-11-03 16:38:51.564151852 +0000 UTC m=+1426.897365422" Nov 03 16:38:51.650467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.647831 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:51.650467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:51.647890 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:52.201654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:52.201600 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:38:52.478795 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:38:52.478757 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:38:52.650390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:52.649902 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Nov 03 16:38:52.650390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:52.649990 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Nov 03 16:38:54.051426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:54.051389 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:54.051938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:54.051788 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:38:54.052742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:38:54.052720 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 16:39:00.198609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:00.198564 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:00.211439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:00.211409 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:00.651095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:00.651058 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:01.256040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.255998 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:39:01.256620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.256605 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8495e090-6715-4250-8022-11f0e88858c4" containerName="revoke-keys" Nov 03 16:39:01.256672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.256621 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8495e090-6715-4250-8022-11f0e88858c4" containerName="revoke-keys" Nov 03 16:39:01.256749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.256739 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8495e090-6715-4250-8022-11f0e88858c4" containerName="revoke-keys" Nov 03 16:39:01.260418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.260399 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.299648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.299609 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:39:01.382492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382433 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382557 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382615 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmcv\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382638 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382789 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382854 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.382965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382951 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.383000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.382974 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.383037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.383007 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484187 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484248 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484292 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmcv\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484314 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484390 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484486 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484618 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484624 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484658 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484725 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.484976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484952 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.485045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.484994 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.485130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.485110 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.485237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.485214 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.486684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.486664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.487045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.487026 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.508793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.508716 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.508921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.508890 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmcv\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.574250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.574214 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:01.892204 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:01.892165 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b713fe4_991b_4e13_be87_4bdab0c74e69.slice/crio-2ae9dce01dc364b5f5d39ec3b85cedf3e2cf92167b4adbd8579d36614e0ab206 WatchSource:0}: Error finding container 2ae9dce01dc364b5f5d39ec3b85cedf3e2cf92167b4adbd8579d36614e0ab206: Status 404 returned error can't find the container with id 2ae9dce01dc364b5f5d39ec3b85cedf3e2cf92167b4adbd8579d36614e0ab206 Nov 03 16:39:01.892385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.892361 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:39:01.894554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.894527 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:01.894641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.894604 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:01.894641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:01.894634 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:02.647467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.645191 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv"] Nov 03 16:39:02.658474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.650669 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.690631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.690582 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:39:02.690989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.690683 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:39:02.690989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.690586 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:02.690989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.690774 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:02.694302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.694271 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv"] Nov 03 16:39:02.705135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.705090 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" event={"ID":"8b713fe4-991b-4e13-be87-4bdab0c74e69","Type":"ContainerStarted","Data":"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771"} Nov 03 16:39:02.705410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.705390 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" event={"ID":"8b713fe4-991b-4e13-be87-4bdab0c74e69","Type":"ContainerStarted","Data":"2ae9dce01dc364b5f5d39ec3b85cedf3e2cf92167b4adbd8579d36614e0ab206"} Nov 03 16:39:02.777282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.776418 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podStartSLOduration=1.776404228 podStartE2EDuration="1.776404228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:02.774602843 +0000 UTC m=+1438.107816413" watchObservedRunningTime="2026-11-03 16:39:02.776404228 +0000 UTC m=+1438.109617797" Nov 03 16:39:02.811290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.811249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.811524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.811507 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsdhn\" (UniqueName: \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.811711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.811697 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.912605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.912513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.912760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.912612 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsdhn\" (UniqueName: \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.912798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.912757 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.913348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.913328 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.914871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.914846 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.941233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.941204 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsdhn\" (UniqueName: \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\") pod \"maas-api-revoke-sub-526f23b9-9c9gv\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:02.983726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:02.983694 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:03.320614 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:03.320578 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232c95d0_49e5_4687_a79f_e750626b221b.slice/crio-fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6 WatchSource:0}: Error finding container fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6: Status 404 returned error can't find the container with id fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6 Nov 03 16:39:03.321025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:03.321005 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv"] Nov 03 16:39:03.574421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:03.574344 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:03.576160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:03.576138 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:03.576266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:03.576185 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:03.713515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:03.713477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" event={"ID":"232c95d0-49e5-4687-a79f-e750626b221b","Type":"ContainerStarted","Data":"fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6"} Nov 03 16:39:04.575350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:04.575316 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:04.575544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:04.575371 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:04.721920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:04.721831 6001 generic.go:356] "Generic (PLEG): container finished" podID="232c95d0-49e5-4687-a79f-e750626b221b" containerID="86971b05269bf5201af64fde131ddaf2608b3eb93272eda1e34d4badb4f38e6e" exitCode=0 Nov 03 16:39:04.721920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:04.721910 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" event={"ID":"232c95d0-49e5-4687-a79f-e750626b221b","Type":"ContainerDied","Data":"86971b05269bf5201af64fde131ddaf2608b3eb93272eda1e34d4badb4f38e6e"} Nov 03 16:39:05.576300 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:05.576264 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:05.576646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:05.576615 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:05.984161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:05.984124 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:06.080820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.080771 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\") pod \"232c95d0-49e5-4687-a79f-e750626b221b\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " Nov 03 16:39:06.081018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.080939 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\") pod \"232c95d0-49e5-4687-a79f-e750626b221b\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " Nov 03 16:39:06.081018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.080980 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\" (UniqueName: \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\") pod \"232c95d0-49e5-4687-a79f-e750626b221b\" (UID: \"232c95d0-49e5-4687-a79f-e750626b221b\") " Nov 03 16:39:06.081631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.081499 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca" pod "232c95d0-49e5-4687-a79f-e750626b221b" (UID: "232c95d0-49e5-4687-a79f-e750626b221b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:06.081631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.081611 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:06.083989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.083958 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca" pod "232c95d0-49e5-4687-a79f-e750626b221b" (UID: "232c95d0-49e5-4687-a79f-e750626b221b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:06.084564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.084526 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn" pod "232c95d0-49e5-4687-a79f-e750626b221b" (UID: "232c95d0-49e5-4687-a79f-e750626b221b"). InnerVolumeSpecName "kube-api-access-lsdhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:06.182529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.182492 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsdhn\" (UniqueName: \"kubernetes.io/projected/232c95d0-49e5-4687-a79f-e750626b221b-kube-api-access-lsdhn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:06.182529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.182527 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/232c95d0-49e5-4687-a79f-e750626b221b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:06.577471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.576931 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:06.577471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.576998 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:06.759487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.752722 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" event={"ID":"232c95d0-49e5-4687-a79f-e750626b221b","Type":"ContainerDied","Data":"fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6"} Nov 03 16:39:06.759487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.752790 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fce545ba8002009cb96f30b829a1d7177aa8bf172c9e32f893e3af198a11b2f6" Nov 03 16:39:06.759487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:06.752866 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv" Nov 03 16:39:07.577894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:07.577853 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:07.579475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:07.579418 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:08.583147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:08.583110 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Nov 03 16:39:08.584095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:08.584059 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Nov 03 16:39:09.579198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:09.579166 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:09.579490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:09.579458 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:09.580137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:09.580119 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:39:15.404641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.404605 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 16:39:15.405409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.405389 6001 cpu_manager.go:414] "containerMap: removing container" podUID="232c95d0-49e5-4687-a79f-e750626b221b" containerName="revoke-keys" Nov 03 16:39:15.405547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.405411 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="232c95d0-49e5-4687-a79f-e750626b221b" containerName="revoke-keys" Nov 03 16:39:15.405599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.405578 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="232c95d0-49e5-4687-a79f-e750626b221b" containerName="revoke-keys" Nov 03 16:39:15.409535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.409503 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.427102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.427065 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn"] Nov 03 16:39:15.432653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.432610 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.447154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.447119 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 16:39:15.450294 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.450261 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:39:15.450871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.450833 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:15.451009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.450926 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:39:15.455611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.455579 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn"] Nov 03 16:39:15.534710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534668 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.534900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534741 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.534900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534865 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.534992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534906 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.534992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534933 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.534992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.534965 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfzf\" (UniqueName: \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.535108 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535071 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.535155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535110 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqzl\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.535155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535133 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.535240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535217 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.535281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535251 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.535331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.535280 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.549560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.549526 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:15.636155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636117 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.636329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636191 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636239 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636269 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636296 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636328 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfzf\" (UniqueName: \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.636543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636374 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636425 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqzl\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636505 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636559 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636619 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.636736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636677 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.636736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.636684 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.637050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.637024 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.637050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.637041 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.637169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.637137 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.637497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.637471 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.637604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.637526 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.638912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.638887 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.639022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.639007 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.639323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.639303 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.660834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.660750 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.661154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.661132 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqzl\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.661247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.661209 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfzf\" (UniqueName: \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\") pod \"maas-api-revoke-sub-ac4787b7-lkjpn\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.729824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.729788 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:15.759973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.759936 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:15.849565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.849315 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt"] Nov 03 16:39:15.855958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.855934 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:15.878892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:15.878861 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt"] Nov 03 16:39:16.041484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.041433 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.041670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.041569 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.041742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.041728 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nxb\" (UniqueName: \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.054293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.054268 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 16:39:16.054699 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:16.054677 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771146da_420a_4d6a_8261_0245e0ee27ad.slice/crio-81bf4187a0dd10566dddf98894d53d8da8426389c7397a729838d06bab5e505e WatchSource:0}: Error finding container 81bf4187a0dd10566dddf98894d53d8da8426389c7397a729838d06bab5e505e: Status 404 returned error can't find the container with id 81bf4187a0dd10566dddf98894d53d8da8426389c7397a729838d06bab5e505e Nov 03 16:39:16.057351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.057318 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:16.057464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.057407 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:16.057537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.057496 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:16.091006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.090945 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn"] Nov 03 16:39:16.091909 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:16.091879 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49360ea3_82dc_42ed_8793_3be8ff47b9c9.slice/crio-cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23 WatchSource:0}: Error finding container cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23: Status 404 returned error can't find the container with id cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23 Nov 03 16:39:16.142566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.142536 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nxb\" (UniqueName: \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.142719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.142643 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.142719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.142702 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.143338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.143313 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.145193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.145167 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.167028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.166994 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nxb\" (UniqueName: \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\") pod \"maas-api-revoke-sub-043be1a2-s97vt\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.185440 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.185404 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:16.320505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.320408 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 16:39:16.328380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.327870 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.352012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.351973 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 16:39:16.447930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.447891 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.447935 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448106 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45mn9\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448153 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448278 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448309 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448355 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448509 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.448749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.448565 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.494032 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:16.493979 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c74aaa_e759_4ceb_950d_8d3f2a8b3895.slice/crio-b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3 WatchSource:0}: Error finding container b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3: Status 404 returned error can't find the container with id b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3 Nov 03 16:39:16.495040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.494954 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt"] Nov 03 16:39:16.549479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549409 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45mn9\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549478 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549547 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549572 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549617 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549735 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549792 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549867 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.549970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.549895 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.550126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.550100 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.550504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.550400 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.550801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.550776 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.550949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.550925 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.552018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.551992 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.552213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.552186 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.552656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.552637 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.576212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.576135 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.576339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.576277 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45mn9\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.651229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.651195 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:16.899883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.899838 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" event={"ID":"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895","Type":"ContainerStarted","Data":"b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3"} Nov 03 16:39:16.901881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.901845 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" event={"ID":"771146da-420a-4d6a-8261-0245e0ee27ad","Type":"ContainerStarted","Data":"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571"} Nov 03 16:39:16.902036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.901901 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" event={"ID":"771146da-420a-4d6a-8261-0245e0ee27ad","Type":"ContainerStarted","Data":"81bf4187a0dd10566dddf98894d53d8da8426389c7397a729838d06bab5e505e"} Nov 03 16:39:16.903699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.903672 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" event={"ID":"49360ea3-82dc-42ed-8793-3be8ff47b9c9","Type":"ContainerStarted","Data":"09cc5a7749829e0b17a5991bd56bc226737b5591c513939493be2de8258a91a0"} Nov 03 16:39:16.903824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.903716 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" event={"ID":"49360ea3-82dc-42ed-8793-3be8ff47b9c9","Type":"ContainerStarted","Data":"cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23"} Nov 03 16:39:16.974469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.972826 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" podStartSLOduration=1.972801654 podStartE2EDuration="1.972801654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:16.943082067 +0000 UTC m=+1452.276295642" watchObservedRunningTime="2026-11-03 16:39:16.972801654 +0000 UTC m=+1452.306015224" Nov 03 16:39:16.991635 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:16.991602 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc41d1b4_0f4d_45b7_bb96_cf584b04e721.slice/crio-57047bb39f659868f5485da79975d65dd55b96ab3d12cbe5505512ab92ed58fc WatchSource:0}: Error finding container 57047bb39f659868f5485da79975d65dd55b96ab3d12cbe5505512ab92ed58fc: Status 404 returned error can't find the container with id 57047bb39f659868f5485da79975d65dd55b96ab3d12cbe5505512ab92ed58fc Nov 03 16:39:16.992257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.992230 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 16:39:16.994149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.994120 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:16.994219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.994181 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:16.994219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:16.994213 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:17.073605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.073538 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podStartSLOduration=2.073517745 podStartE2EDuration="2.073517745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:17.014708981 +0000 UTC m=+1452.347922549" watchObservedRunningTime="2026-11-03 16:39:17.073517745 +0000 UTC m=+1452.406731316" Nov 03 16:39:17.218489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.218409 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" podStartSLOduration=2.218393295 podStartE2EDuration="2.218393295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:17.069606181 +0000 UTC m=+1452.402819751" watchObservedRunningTime="2026-11-03 16:39:17.218393295 +0000 UTC m=+1452.551606864" Nov 03 16:39:17.219249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.219218 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 16:39:17.223900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.223874 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.292499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.291078 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 16:39:17.386785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.386752 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.386999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.386907 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.386999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.386966 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387212 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387246 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387263 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387341 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387416 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.387521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.387502 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7pq\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489028 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489088 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489135 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489174 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489198 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489235 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489312 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489347 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7pq\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489422 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489841 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.489932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.489897 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.490136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.490108 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.490418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.490395 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.490541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.490469 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.492678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.492649 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.493138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.493111 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.522563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.522531 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.522760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.522686 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7pq\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.592630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.592591 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:17.730263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.730198 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:17.731355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.731328 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:17.731720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.731378 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:17.921607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.920895 6001 generic.go:356] "Generic (PLEG): container finished" podID="d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" containerID="ee974a0b360e91b6b44a0d45fd8475c61d2873bc0a988f4a8a57d71abc3b33e1" exitCode=0 Nov 03 16:39:17.921607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.921127 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" event={"ID":"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895","Type":"ContainerDied","Data":"ee974a0b360e91b6b44a0d45fd8475c61d2873bc0a988f4a8a57d71abc3b33e1"} Nov 03 16:39:17.928908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.928533 6001 generic.go:356] "Generic (PLEG): container finished" podID="49360ea3-82dc-42ed-8793-3be8ff47b9c9" containerID="09cc5a7749829e0b17a5991bd56bc226737b5591c513939493be2de8258a91a0" exitCode=0 Nov 03 16:39:17.928908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.928659 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" event={"ID":"49360ea3-82dc-42ed-8793-3be8ff47b9c9","Type":"ContainerDied","Data":"09cc5a7749829e0b17a5991bd56bc226737b5591c513939493be2de8258a91a0"} Nov 03 16:39:17.933471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.933409 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" event={"ID":"cc41d1b4-0f4d-45b7-bb96-cf584b04e721","Type":"ContainerStarted","Data":"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782"} Nov 03 16:39:17.933723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.933697 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" event={"ID":"cc41d1b4-0f4d-45b7-bb96-cf584b04e721","Type":"ContainerStarted","Data":"57047bb39f659868f5485da79975d65dd55b96ab3d12cbe5505512ab92ed58fc"} Nov 03 16:39:17.990209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:17.988713 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 16:39:18.009134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.009098 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:18.009333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.009316 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:18.009472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.009455 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:18.034365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.034309 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podStartSLOduration=2.034295543 podStartE2EDuration="2.034295543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:18.032223338 +0000 UTC m=+1453.365436907" watchObservedRunningTime="2026-11-03 16:39:18.034295543 +0000 UTC m=+1453.367509111" Nov 03 16:39:18.653879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.653837 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:18.660638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.660599 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.660905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.660870 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:18.731067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.731032 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.731351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.731310 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:18.982540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.967510 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" event={"ID":"c1336b33-4119-41cf-b0bd-6458d1ead764","Type":"ContainerStarted","Data":"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458"} Nov 03 16:39:18.982540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:18.967573 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" event={"ID":"c1336b33-4119-41cf-b0bd-6458d1ead764","Type":"ContainerStarted","Data":"f74c77964380b58c6775a8ceff2d6e54413edc554c5cd569b937f73d5960d469"} Nov 03 16:39:19.027839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.027773 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podStartSLOduration=2.027751808 podStartE2EDuration="2.027751808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:19.024905683 +0000 UTC m=+1454.358119254" watchObservedRunningTime="2026-11-03 16:39:19.027751808 +0000 UTC m=+1454.360965378" Nov 03 16:39:19.605286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.605236 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:19.635294 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.635251 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.636427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.636390 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:19.654311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.654251 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.671616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.671576 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:19.734481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.732571 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.734481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:19.732638 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:20.495141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.495107 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:20.611540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.596463 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.611540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.596538 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:20.660674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.659714 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" event={"ID":"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895","Type":"ContainerDied","Data":"b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3"} Nov 03 16:39:20.660674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.659833 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4eef4aed135a5a7e58e78c55b6dfbe0e13bad659ed4c5aedf040e3a0fe60cc3" Nov 03 16:39:20.719501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.686722 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\" (UniqueName: \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\") pod \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " Nov 03 16:39:20.719501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.686918 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\") pod \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " Nov 03 16:39:20.719501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.686970 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\") pod \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\" (UID: \"d0c74aaa-e759-4ceb-950d-8d3f2a8b3895\") " Nov 03 16:39:20.719501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.718953 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.719501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.719039 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:20.732516 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.725064 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:20.780471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.752004 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca" pod "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" (UID: "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:20.841972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.841934 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.842308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.842277 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:20.907528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.882079 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\") pod \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " Nov 03 16:39:20.922470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.910733 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\") pod \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " Nov 03 16:39:20.922984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.922956 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\" (UniqueName: \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\") pod \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\" (UID: \"49360ea3-82dc-42ed-8793-3be8ff47b9c9\") " Nov 03 16:39:20.989248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:20.989207 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.058028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.054035 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.062430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.062215 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca" pod "49360ea3-82dc-42ed-8793-3be8ff47b9c9" (UID: "49360ea3-82dc-42ed-8793-3be8ff47b9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:21.076037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.075994 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf" pod "49360ea3-82dc-42ed-8793-3be8ff47b9c9" (UID: "49360ea3-82dc-42ed-8793-3be8ff47b9c9"). InnerVolumeSpecName "kube-api-access-2mfzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:21.098538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.097741 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfzf\" (UniqueName: \"kubernetes.io/projected/49360ea3-82dc-42ed-8793-3be8ff47b9c9-kube-api-access-2mfzf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.098538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.097785 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.171671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.171541 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:39:21.242477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.237836 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb" pod "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" (UID: "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895"). InnerVolumeSpecName "kube-api-access-f5nxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:21.250992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.250946 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nxb\" (UniqueName: \"kubernetes.io/projected/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-kube-api-access-f5nxb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.284541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.274805 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca" pod "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" (UID: "d0c74aaa-e759-4ceb-950d-8d3f2a8b3895"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:21.285261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.285225 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca" pod "49360ea3-82dc-42ed-8793-3be8ff47b9c9" (UID: "49360ea3-82dc-42ed-8793-3be8ff47b9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:21.351472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.350530 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.351472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.350893 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:21.366703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.366638 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.388676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.388637 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/49360ea3-82dc-42ed-8793-3be8ff47b9c9-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.388932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.388916 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:21.614470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.613531 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.614470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.613601 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:21.653482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.653009 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.653482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.653072 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:21.690470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.689913 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 16:39:21.742467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.741853 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.742467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.741963 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:21.897509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:21.895635 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.020606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.020545 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.056614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.056112 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:22.084157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.084121 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:39:22.193456 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.191020 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.194393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.194367 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:39:22.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.267633 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" Nov 03 16:39:22.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.348130 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn" event={"ID":"49360ea3-82dc-42ed-8793-3be8ff47b9c9","Type":"ContainerDied","Data":"cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23"} Nov 03 16:39:22.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.348194 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfbd8f54a948cdd7bc327a7cebe8610e55d30bdcc5b13df3eb65b0f1c7a1ce23" Nov 03 16:39:22.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.348270 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt" Nov 03 16:39:22.384957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.384914 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:39:22.463581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.396210 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:39:22.463581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.429069 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:22.463581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.429610 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:39:22.661057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.661015 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.661354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.661319 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:22.705470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.680672 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.706121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.706081 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:22.706303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.706282 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.706945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.706922 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:22.762992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.762954 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.763250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.763222 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:22.902681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:22.902630 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.068587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.059012 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:39:23.087659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.087599 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.200753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.200716 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:39:23.201069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.201049 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:39:23.205676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.202865 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.387415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.385756 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:39:23.432724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.432679 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.682531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.676912 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.682531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.676977 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:23.718604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.706045 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.718604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.706123 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:23.718604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.706622 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.769678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.751557 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.769678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.751620 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:23.915471 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:39:23.912658 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:39:23.915471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:23.912766 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.039482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.036714 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.120572 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:39:24.111745 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c74aaa_e759_4ceb_950d_8d3f2a8b3895.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49360ea3_82dc_42ed_8793_3be8ff47b9c9.slice\": RecentStats: unable to find data in memory cache]" Nov 03 16:39:24.185998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.133479 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:39:24.185998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.135973 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.238054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.238010 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:39:24.477535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.393416 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.503801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.503753 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:24.614600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.603316 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.614600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.603387 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:24.655971 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.655928 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.656262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.656227 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:24.740528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.739738 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.805725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.805682 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.805992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.805957 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:24.834460 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.834222 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:24.834764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.834746 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:39:24.834872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.834863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:39:24.834977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.834968 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:39:24.837631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.837603 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:39:24.843474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.838608 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:39:24.843474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:24.841357 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:39:25.593016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.592953 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.593231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.593021 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:25.660469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.656975 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.660469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.657041 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 16:39:25.730613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.730568 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.730828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.730639 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 16:39:25.817467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817065 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm"] Nov 03 16:39:25.817899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817724 6001 cpu_manager.go:414] "containerMap: removing container" podUID="49360ea3-82dc-42ed-8793-3be8ff47b9c9" containerName="revoke-keys" Nov 03 16:39:25.817899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817738 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49360ea3-82dc-42ed-8793-3be8ff47b9c9" containerName="revoke-keys" Nov 03 16:39:25.817899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817788 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" containerName="revoke-keys" Nov 03 16:39:25.817899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817794 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" containerName="revoke-keys" Nov 03 16:39:25.817899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817903 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" containerName="revoke-keys" Nov 03 16:39:25.818079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.817914 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="49360ea3-82dc-42ed-8793-3be8ff47b9c9" containerName="revoke-keys" Nov 03 16:39:25.826480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.824077 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:25.859530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.857592 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:25.859530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.857852 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:39:25.859530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.857949 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:25.859530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.858017 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:39:25.891486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.888889 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm"] Nov 03 16:39:25.942154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.942107 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:25.942372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.942280 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qckx\" (UniqueName: \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:25.942372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:25.942371 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.044373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.044325 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qckx\" (UniqueName: \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.044841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.044813 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.051643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.051600 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.052513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.045778 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.077471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.073937 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.080727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.080694 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qckx\" (UniqueName: \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\") pod \"maas-api-revoke-sub-cf151c15-k9qpm\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.212386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.212022 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:26.220575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.218868 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz"] Nov 03 16:39:26.225839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.225019 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.300495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.276186 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz"] Nov 03 16:39:26.399746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.399311 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.399746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.399403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgt6\" (UniqueName: \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.399746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.399485 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.501425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.501378 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.501607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.501524 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgt6\" (UniqueName: \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.501650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.501624 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.502678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.502642 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.510215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.509299 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.539793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.539735 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgt6\" (UniqueName: \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\") pod \"maas-api-revoke-sub-36a6b7e5-86dnz\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.593589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.593079 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:26.593589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.593142 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:26.609241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.608871 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:26.815568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.815530 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm"] Nov 03 16:39:26.815787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.815768 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:26.818223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.818194 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:26.829334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.829288 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 16:39:26.830813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.830783 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" event={"ID":"8be00747-f424-4825-b8d6-4f9ba7e1a571","Type":"ContainerStarted","Data":"52f5ad2f31437630910cef634410569670a3c8a45bd92587b7c7a17b8d8f8fb9"} Nov 03 16:39:26.974541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:26.974499 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz"] Nov 03 16:39:26.984638 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:26.984594 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0984622a_9487_4288_8639_262d4e08f2e5.slice/crio-428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b WatchSource:0}: Error finding container 428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b: Status 404 returned error can't find the container with id 428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b Nov 03 16:39:27.593990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.593891 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 16:39:27.593990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.593969 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 16:39:27.708765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.708705 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:27.709027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.708817 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:27.842299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.842261 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" event={"ID":"8be00747-f424-4825-b8d6-4f9ba7e1a571","Type":"ContainerStarted","Data":"1e4578f6194803bc18f25daa5adf6a0b11d4697f674ae96209d8ff5fe7a60811"} Nov 03 16:39:27.846604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.846495 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" event={"ID":"0984622a-9487-4288-8639-262d4e08f2e5","Type":"ContainerStarted","Data":"d7fb0ab5c9a81e4c7dc9e283bf60e1697692f723ae40c49872c3e75d10bf7c2b"} Nov 03 16:39:27.846604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.846556 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" event={"ID":"0984622a-9487-4288-8639-262d4e08f2e5","Type":"ContainerStarted","Data":"428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b"} Nov 03 16:39:27.896798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.896654 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" podStartSLOduration=2.896632492 podStartE2EDuration="2.896632492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:27.890999521 +0000 UTC m=+1463.224213085" watchObservedRunningTime="2026-11-03 16:39:27.896632492 +0000 UTC m=+1463.229846062" Nov 03 16:39:27.937650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:27.937581 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" podStartSLOduration=1.937557094 podStartE2EDuration="1.937557094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:27.934412961 +0000 UTC m=+1463.267626530" watchObservedRunningTime="2026-11-03 16:39:27.937557094 +0000 UTC m=+1463.270770664" Nov 03 16:39:28.422339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.422304 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:28.422763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.422739 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:28.423517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.423491 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 16:39:28.853827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.853793 6001 generic.go:356] "Generic (PLEG): container finished" podID="8be00747-f424-4825-b8d6-4f9ba7e1a571" containerID="1e4578f6194803bc18f25daa5adf6a0b11d4697f674ae96209d8ff5fe7a60811" exitCode=0 Nov 03 16:39:28.854258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.853872 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" event={"ID":"8be00747-f424-4825-b8d6-4f9ba7e1a571","Type":"ContainerDied","Data":"1e4578f6194803bc18f25daa5adf6a0b11d4697f674ae96209d8ff5fe7a60811"} Nov 03 16:39:28.855672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.855645 6001 generic.go:356] "Generic (PLEG): container finished" podID="0984622a-9487-4288-8639-262d4e08f2e5" containerID="d7fb0ab5c9a81e4c7dc9e283bf60e1697692f723ae40c49872c3e75d10bf7c2b" exitCode=0 Nov 03 16:39:28.855782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:28.855731 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" event={"ID":"0984622a-9487-4288-8639-262d4e08f2e5","Type":"ContainerDied","Data":"d7fb0ab5c9a81e4c7dc9e283bf60e1697692f723ae40c49872c3e75d10bf7c2b"} Nov 03 16:39:29.593883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:29.593844 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:29.594108 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:29.593901 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:29.633070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:29.633036 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:29.633505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:29.633482 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:29.634615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:29.634592 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 16:39:30.054724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.054694 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:30.109834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.109805 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:30.195892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.195797 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\") pod \"0984622a-9487-4288-8639-262d4e08f2e5\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " Nov 03 16:39:30.196080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.195923 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\" (UniqueName: \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\") pod \"0984622a-9487-4288-8639-262d4e08f2e5\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " Nov 03 16:39:30.196080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.196046 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\") pod \"0984622a-9487-4288-8639-262d4e08f2e5\" (UID: \"0984622a-9487-4288-8639-262d4e08f2e5\") " Nov 03 16:39:30.196453 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.196404 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca" pod "0984622a-9487-4288-8639-262d4e08f2e5" (UID: "0984622a-9487-4288-8639-262d4e08f2e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:30.196698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.196675 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.198136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.198105 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca" pod "0984622a-9487-4288-8639-262d4e08f2e5" (UID: "0984622a-9487-4288-8639-262d4e08f2e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:30.199484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.198460 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6" pod "0984622a-9487-4288-8639-262d4e08f2e5" (UID: "0984622a-9487-4288-8639-262d4e08f2e5"). InnerVolumeSpecName "kube-api-access-wzgt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:30.297948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.297917 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\") pod \"8be00747-f424-4825-b8d6-4f9ba7e1a571\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " Nov 03 16:39:30.298129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.297964 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\" (UniqueName: \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\") pod \"8be00747-f424-4825-b8d6-4f9ba7e1a571\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " Nov 03 16:39:30.298129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.298002 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\") pod \"8be00747-f424-4825-b8d6-4f9ba7e1a571\" (UID: \"8be00747-f424-4825-b8d6-4f9ba7e1a571\") " Nov 03 16:39:30.298395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.298366 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca" pod "8be00747-f424-4825-b8d6-4f9ba7e1a571" (UID: "8be00747-f424-4825-b8d6-4f9ba7e1a571"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:30.298844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.298815 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgt6\" (UniqueName: \"kubernetes.io/projected/0984622a-9487-4288-8639-262d4e08f2e5-kube-api-access-wzgt6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.298844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.298845 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.299034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.298861 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0984622a-9487-4288-8639-262d4e08f2e5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.300237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.300211 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx" pod "8be00747-f424-4825-b8d6-4f9ba7e1a571" (UID: "8be00747-f424-4825-b8d6-4f9ba7e1a571"). InnerVolumeSpecName "kube-api-access-5qckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:30.300640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.300614 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca" pod "8be00747-f424-4825-b8d6-4f9ba7e1a571" (UID: "8be00747-f424-4825-b8d6-4f9ba7e1a571"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:30.401221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.401187 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be00747-f424-4825-b8d6-4f9ba7e1a571-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.401221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.401221 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qckx\" (UniqueName: \"kubernetes.io/projected/8be00747-f424-4825-b8d6-4f9ba7e1a571-kube-api-access-5qckx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:30.873923 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.873883 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" Nov 03 16:39:30.874083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.873894 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm" event={"ID":"8be00747-f424-4825-b8d6-4f9ba7e1a571","Type":"ContainerDied","Data":"52f5ad2f31437630910cef634410569670a3c8a45bd92587b7c7a17b8d8f8fb9"} Nov 03 16:39:30.874083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.874035 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52f5ad2f31437630910cef634410569670a3c8a45bd92587b7c7a17b8d8f8fb9" Nov 03 16:39:30.875499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.875437 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" Nov 03 16:39:30.875499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.875475 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz" event={"ID":"0984622a-9487-4288-8639-262d4e08f2e5","Type":"ContainerDied","Data":"428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b"} Nov 03 16:39:30.875703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:30.875545 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="428e5ce82829073bbcf91b5d039a4e3f2aec97bf4e039e5dd4282b165914159b" Nov 03 16:39:36.243490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.243439 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:39:36.244144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244128 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0984622a-9487-4288-8639-262d4e08f2e5" containerName="revoke-keys" Nov 03 16:39:36.244144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244144 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0984622a-9487-4288-8639-262d4e08f2e5" containerName="revoke-keys" Nov 03 16:39:36.244259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244168 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8be00747-f424-4825-b8d6-4f9ba7e1a571" containerName="revoke-keys" Nov 03 16:39:36.244259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244173 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be00747-f424-4825-b8d6-4f9ba7e1a571" containerName="revoke-keys" Nov 03 16:39:36.244359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244270 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0984622a-9487-4288-8639-262d4e08f2e5" containerName="revoke-keys" Nov 03 16:39:36.244359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.244283 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be00747-f424-4825-b8d6-4f9ba7e1a571" containerName="revoke-keys" Nov 03 16:39:36.249046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.249023 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.284500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.284458 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:39:36.327289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327253 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327292 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327406 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327508 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327540 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdhhs\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327630 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327709 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.327893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.328098 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.327974 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.429619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429582 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.429837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429719 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.429837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429748 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.429837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429801 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429838 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429874 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdhhs\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429928 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.429957 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.430064 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.430336 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.430436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.430413 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.431024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.430983 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.431024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.431018 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.431260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.431239 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.433518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.433492 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.433935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.433911 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.455641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.455605 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.455817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.455683 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdhhs\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.567871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.567778 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:36.900631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.900593 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:39:36.900948 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:36.900923 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48dcf133_89d9_4315_b38c_268f6740f920.slice/crio-0a35d4399d31e5416b70189ee055fac7ce535388cff763e0b43c79738c60c535 WatchSource:0}: Error finding container 0a35d4399d31e5416b70189ee055fac7ce535388cff763e0b43c79738c60c535: Status 404 returned error can't find the container with id 0a35d4399d31e5416b70189ee055fac7ce535388cff763e0b43c79738c60c535 Nov 03 16:39:36.903848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.903812 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:36.903961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.903882 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:36.903961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:36.903909 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:37.022526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.022477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" event={"ID":"48dcf133-89d9-4315-b38c-268f6740f920","Type":"ContainerStarted","Data":"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343"} Nov 03 16:39:37.022753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.022589 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" event={"ID":"48dcf133-89d9-4315-b38c-268f6740f920","Type":"ContainerStarted","Data":"0a35d4399d31e5416b70189ee055fac7ce535388cff763e0b43c79738c60c535"} Nov 03 16:39:37.072495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.072273 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podStartSLOduration=1.072259834 podStartE2EDuration="1.072259834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:37.071827999 +0000 UTC m=+1472.405041571" watchObservedRunningTime="2026-11-03 16:39:37.072259834 +0000 UTC m=+1472.405473404" Nov 03 16:39:37.571472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.571151 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:37.571472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.571295 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:37.571472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:37.571344 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:38.568925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:38.568880 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:38.569112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:38.568945 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:38.976030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:38.975990 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp"] Nov 03 16:39:38.980501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:38.980478 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:38.994039 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:38.994007 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp"] Nov 03 16:39:39.002271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.002238 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:39.002271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.002240 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:39:39.002271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.002274 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:39:39.009103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.009075 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:39.085261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.085220 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.085484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.085377 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.085484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.085427 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-268zq\" (UniqueName: \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.187405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.187359 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.187580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.187514 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.187580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.187569 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-268zq\" (UniqueName: \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.188266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.188237 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.190038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.190013 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.212821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.212778 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-268zq\" (UniqueName: \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\") pod \"maas-api-revoke-sub-b7907f6a-j5vcp\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.298791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.298688 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:39.568985 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.568896 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:39.568985 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.568958 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:39.637735 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:39.637692 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee62d86_cdad_4f50_8279_d9c2646b765c.slice/crio-e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128 WatchSource:0}: Error finding container e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128: Status 404 returned error can't find the container with id e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128 Nov 03 16:39:39.640663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:39.640628 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp"] Nov 03 16:39:40.070772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:40.070685 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" event={"ID":"8ee62d86-cdad-4f50-8279-d9c2646b765c","Type":"ContainerStarted","Data":"e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128"} Nov 03 16:39:40.568938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:40.568892 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:40.569165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:40.568969 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:41.079236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:41.079198 6001 generic.go:356] "Generic (PLEG): container finished" podID="8ee62d86-cdad-4f50-8279-d9c2646b765c" containerID="27c553cb34205e72968d123122b118fb7e32727246ed28d76a9444e874280606" exitCode=0 Nov 03 16:39:41.079683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:41.079278 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" event={"ID":"8ee62d86-cdad-4f50-8279-d9c2646b765c","Type":"ContainerDied","Data":"27c553cb34205e72968d123122b118fb7e32727246ed28d76a9444e874280606"} Nov 03 16:39:41.568881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:41.568847 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:41.569070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:41.568905 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:42.282844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.282813 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:42.434883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.434797 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\" (UniqueName: \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\") pod \"8ee62d86-cdad-4f50-8279-d9c2646b765c\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " Nov 03 16:39:42.434883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.434836 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\") pod \"8ee62d86-cdad-4f50-8279-d9c2646b765c\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " Nov 03 16:39:42.434883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.434883 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\") pod \"8ee62d86-cdad-4f50-8279-d9c2646b765c\" (UID: \"8ee62d86-cdad-4f50-8279-d9c2646b765c\") " Nov 03 16:39:42.435243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.435218 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca" pod "8ee62d86-cdad-4f50-8279-d9c2646b765c" (UID: "8ee62d86-cdad-4f50-8279-d9c2646b765c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:42.435367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.435352 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:42.437146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.437121 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca" pod "8ee62d86-cdad-4f50-8279-d9c2646b765c" (UID: "8ee62d86-cdad-4f50-8279-d9c2646b765c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:42.437146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.437122 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq" pod "8ee62d86-cdad-4f50-8279-d9c2646b765c" (UID: "8ee62d86-cdad-4f50-8279-d9c2646b765c"). InnerVolumeSpecName "kube-api-access-268zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:42.536505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.536467 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-268zq\" (UniqueName: \"kubernetes.io/projected/8ee62d86-cdad-4f50-8279-d9c2646b765c-kube-api-access-268zq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:42.536505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.536501 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee62d86-cdad-4f50-8279-d9c2646b765c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:42.569701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.569657 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:42.569911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.569724 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:42.715595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.715463 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr"] Nov 03 16:39:42.716259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.716238 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8ee62d86-cdad-4f50-8279-d9c2646b765c" containerName="revoke-keys" Nov 03 16:39:42.716330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.716260 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee62d86-cdad-4f50-8279-d9c2646b765c" containerName="revoke-keys" Nov 03 16:39:42.716388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.716379 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee62d86-cdad-4f50-8279-d9c2646b765c" containerName="revoke-keys" Nov 03 16:39:42.720202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.720176 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.766473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.764704 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr"] Nov 03 16:39:42.841086 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.841047 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr847\" (UniqueName: \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.841086 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.841085 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.841506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.841474 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.942524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.942483 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.942720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.942608 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr847\" (UniqueName: \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.942720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.942632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.943137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.943114 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.969587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.969423 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:42.969743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:42.969648 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr847\" (UniqueName: \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\") pod \"maas-api-revoke-sub-2b62b149-r6bcr\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:43.036879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.036826 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:43.103713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.103300 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" event={"ID":"8ee62d86-cdad-4f50-8279-d9c2646b765c","Type":"ContainerDied","Data":"e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128"} Nov 03 16:39:43.103713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.103374 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e80761e04ff5891fc11dce63a6f00b8ab008be231f81161672f32a8600f09128" Nov 03 16:39:43.103713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.103473 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp" Nov 03 16:39:43.394099 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:43.394027 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f3f7cb_0764_4c77_8380_afba76c25f7c.slice/crio-7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f WatchSource:0}: Error finding container 7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f: Status 404 returned error can't find the container with id 7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f Nov 03 16:39:43.394969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.394942 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr"] Nov 03 16:39:43.568711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.568679 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 16:39:43.568870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:43.568740 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 16:39:44.114624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.114501 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" event={"ID":"d9f3f7cb-0764-4c77-8380-afba76c25f7c","Type":"ContainerStarted","Data":"09a727110d4e679817b700f7a2b226881ca5fb28f14024f78422ff028cd54aeb"} Nov 03 16:39:44.114624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.114582 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" event={"ID":"d9f3f7cb-0764-4c77-8380-afba76c25f7c","Type":"ContainerStarted","Data":"7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f"} Nov 03 16:39:44.184217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.184132 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" podStartSLOduration=2.184111773 podStartE2EDuration="2.184111773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:44.183796064 +0000 UTC m=+1479.517009634" watchObservedRunningTime="2026-11-03 16:39:44.184111773 +0000 UTC m=+1479.517325345" Nov 03 16:39:44.478861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.478822 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:39:44.483261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.483239 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.541505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.535133 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:39:44.566679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.566621 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.566873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.566832 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.566980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.566960 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567030 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567072 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567239 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567317 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567293 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567397 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.567512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.567433 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jklg\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.573139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.573107 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:44.573577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.573556 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:44.574288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.574266 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:39:44.668741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.668703 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.668958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.668760 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.668958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.668939 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jklg\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669023 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669169 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669199 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669177 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669276 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669317 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669355 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669518 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669727 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.669908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.669872 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.670214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.670191 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.671981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.671960 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.672113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.671979 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.724438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.699434 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jklg\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.724438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.699708 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:44.801179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:44.801089 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:45.064296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.064207 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr"] Nov 03 16:39:45.069957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.069926 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.090522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.090375 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr"] Nov 03 16:39:45.125964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.125930 6001 generic.go:356] "Generic (PLEG): container finished" podID="d9f3f7cb-0764-4c77-8380-afba76c25f7c" containerID="09a727110d4e679817b700f7a2b226881ca5fb28f14024f78422ff028cd54aeb" exitCode=0 Nov 03 16:39:45.126124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.126008 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" event={"ID":"d9f3f7cb-0764-4c77-8380-afba76c25f7c","Type":"ContainerDied","Data":"09a727110d4e679817b700f7a2b226881ca5fb28f14024f78422ff028cd54aeb"} Nov 03 16:39:45.145949 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:45.145918 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29009892_ebc1_40ca_b2b3_4ccc3fd554aa.slice/crio-2f6655bfd28e32765ba4e61c8e3b71d616b756b703a3d4c68d8e080d40caf5ff WatchSource:0}: Error finding container 2f6655bfd28e32765ba4e61c8e3b71d616b756b703a3d4c68d8e080d40caf5ff: Status 404 returned error can't find the container with id 2f6655bfd28e32765ba4e61c8e3b71d616b756b703a3d4c68d8e080d40caf5ff Nov 03 16:39:45.146841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.146817 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:39:45.148634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.148605 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:45.148701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.148668 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:45.148734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.148699 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:45.178965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.178928 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.179078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.179009 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hgn\" (UniqueName: \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.179266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.179245 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.281327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.281281 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.281567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.281362 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hgn\" (UniqueName: \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.281707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.281679 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.282461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.282422 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.284008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.283984 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.305582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.305542 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hgn\" (UniqueName: \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\") pod \"maas-api-revoke-sub-95bf16e4-zwcbr\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.389585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.389489 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:45.706116 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:45.706083 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d53fd1_efa7_4ca7_b113_d84d18e14038.slice/crio-29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5 WatchSource:0}: Error finding container 29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5: Status 404 returned error can't find the container with id 29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5 Nov 03 16:39:45.707224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:45.707195 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr"] Nov 03 16:39:46.134636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.134598 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" event={"ID":"50d53fd1-efa7-4ca7-b113-d84d18e14038","Type":"ContainerStarted","Data":"29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5"} Nov 03 16:39:46.136296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.136261 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" event={"ID":"29009892-ebc1-40ca-b2b3-4ccc3fd554aa","Type":"ContainerStarted","Data":"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8"} Nov 03 16:39:46.136415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.136322 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" event={"ID":"29009892-ebc1-40ca-b2b3-4ccc3fd554aa","Type":"ContainerStarted","Data":"2f6655bfd28e32765ba4e61c8e3b71d616b756b703a3d4c68d8e080d40caf5ff"} Nov 03 16:39:46.198478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.198390 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podStartSLOduration=2.198367458 podStartE2EDuration="2.198367458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:46.194973848 +0000 UTC m=+1481.528187416" watchObservedRunningTime="2026-11-03 16:39:46.198367458 +0000 UTC m=+1481.531581027" Nov 03 16:39:46.441375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.441345 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:46.604233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.604131 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\") pod \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " Nov 03 16:39:46.604419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.604333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\") pod \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " Nov 03 16:39:46.604419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.604401 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\" (UniqueName: \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\") pod \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\" (UID: \"d9f3f7cb-0764-4c77-8380-afba76c25f7c\") " Nov 03 16:39:46.604728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.604703 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca" pod "d9f3f7cb-0764-4c77-8380-afba76c25f7c" (UID: "d9f3f7cb-0764-4c77-8380-afba76c25f7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:46.604997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.604980 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:46.606379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.606346 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca" pod "d9f3f7cb-0764-4c77-8380-afba76c25f7c" (UID: "d9f3f7cb-0764-4c77-8380-afba76c25f7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:46.606503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.606481 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847" pod "d9f3f7cb-0764-4c77-8380-afba76c25f7c" (UID: "d9f3f7cb-0764-4c77-8380-afba76c25f7c"). InnerVolumeSpecName "kube-api-access-sr847". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:46.705941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.705906 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr847\" (UniqueName: \"kubernetes.io/projected/d9f3f7cb-0764-4c77-8380-afba76c25f7c-kube-api-access-sr847\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:46.705941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.705939 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9f3f7cb-0764-4c77-8380-afba76c25f7c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:46.803721 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.803686 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:46.807147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.807109 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:46.807350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:46.807165 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:47.153083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.152183 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" event={"ID":"d9f3f7cb-0764-4c77-8380-afba76c25f7c","Type":"ContainerDied","Data":"7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f"} Nov 03 16:39:47.153083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.152255 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fcb79fab534ac78863c7ed3f85251a272713dbe0c4c326467a05e30805ed21f" Nov 03 16:39:47.153083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.152329 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr" Nov 03 16:39:47.163884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.162324 6001 generic.go:356] "Generic (PLEG): container finished" podID="50d53fd1-efa7-4ca7-b113-d84d18e14038" containerID="5de119ec9f3b15544bb5d9cf255ff24fa75fb1a4a9bf197c7b6e37bee9115161" exitCode=0 Nov 03 16:39:47.163884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.163720 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" event={"ID":"50d53fd1-efa7-4ca7-b113-d84d18e14038","Type":"ContainerDied","Data":"5de119ec9f3b15544bb5d9cf255ff24fa75fb1a4a9bf197c7b6e37bee9115161"} Nov 03 16:39:47.302301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.302259 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:39:47.303059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.303027 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d9f3f7cb-0764-4c77-8380-afba76c25f7c" containerName="revoke-keys" Nov 03 16:39:47.303059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.303054 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f3f7cb-0764-4c77-8380-afba76c25f7c" containerName="revoke-keys" Nov 03 16:39:47.303248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.303183 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f3f7cb-0764-4c77-8380-afba76c25f7c" containerName="revoke-keys" Nov 03 16:39:47.307047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.307024 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.335004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.334969 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:39:47.415093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415132 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415232 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415367 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415406 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415511 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415624 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415655 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq9lf\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.415776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.415764 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.517326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.517276 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.517542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.517492 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.517656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.517581 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.517861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.517824 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.517956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.517878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.519158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519128 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.519406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519388 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.519645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519628 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.519787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519764 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.519928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519913 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.520012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.519947 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq9lf\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.520066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.520005 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.520122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.520094 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.520748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.520723 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.525499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.525426 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.527431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.527408 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.563525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.545581 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq9lf\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.568467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.565965 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.624332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.624293 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:47.803034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.802011 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:47.803034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.802078 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:47.991482 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:39:47.991437 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0642b25b_8bc6_4918_b666_359d42813315.slice/crio-7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf WatchSource:0}: Error finding container 7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf: Status 404 returned error can't find the container with id 7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf Nov 03 16:39:47.993248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.993227 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:39:47.994148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.994117 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:47.994209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.994178 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:47.994248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:47.994208 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:39:48.182771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.182728 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" event={"ID":"0642b25b-8bc6-4918-b666-359d42813315","Type":"ContainerStarted","Data":"b41f9a3fec6c17fe4a8bddd899fc2d63d630108c3d5322566bd69eda3be581e7"} Nov 03 16:39:48.182932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.182789 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" event={"ID":"0642b25b-8bc6-4918-b666-359d42813315","Type":"ContainerStarted","Data":"7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf"} Nov 03 16:39:48.232937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.232887 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podStartSLOduration=1.232812882 podStartE2EDuration="1.232812882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:48.230060838 +0000 UTC m=+1483.563274433" watchObservedRunningTime="2026-11-03 16:39:48.232812882 +0000 UTC m=+1483.566026526" Nov 03 16:39:48.372559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.367860 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:48.434598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.434551 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\") pod \"50d53fd1-efa7-4ca7-b113-d84d18e14038\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " Nov 03 16:39:48.434809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.434649 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\") pod \"50d53fd1-efa7-4ca7-b113-d84d18e14038\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " Nov 03 16:39:48.434809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.434791 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\" (UniqueName: \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\") pod \"50d53fd1-efa7-4ca7-b113-d84d18e14038\" (UID: \"50d53fd1-efa7-4ca7-b113-d84d18e14038\") " Nov 03 16:39:48.435511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.435426 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca" pod "50d53fd1-efa7-4ca7-b113-d84d18e14038" (UID: "50d53fd1-efa7-4ca7-b113-d84d18e14038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:48.436159 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.436129 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:48.495796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.495749 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn" pod "50d53fd1-efa7-4ca7-b113-d84d18e14038" (UID: "50d53fd1-efa7-4ca7-b113-d84d18e14038"). InnerVolumeSpecName "kube-api-access-69hgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:48.496106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.496077 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca" pod "50d53fd1-efa7-4ca7-b113-d84d18e14038" (UID: "50d53fd1-efa7-4ca7-b113-d84d18e14038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:48.537255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.537158 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50d53fd1-efa7-4ca7-b113-d84d18e14038-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:48.537255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.537190 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hgn\" (UniqueName: \"kubernetes.io/projected/50d53fd1-efa7-4ca7-b113-d84d18e14038-kube-api-access-69hgn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:48.625299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.625258 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:39:48.801738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.801651 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:48.801738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:48.801713 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:49.207560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.207472 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" Nov 03 16:39:49.207938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.207614 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr" event={"ID":"50d53fd1-efa7-4ca7-b113-d84d18e14038","Type":"ContainerDied","Data":"29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5"} Nov 03 16:39:49.207938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.207658 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bbac6f994dc621e7b01c8f95ac751052fd472c4cefcfc13fa336d4617120f5" Nov 03 16:39:49.626423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.626379 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.626627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.626483 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:49.802471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.801499 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.802471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:49.801558 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:50.626153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:50.626124 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.627505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:50.627261 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:50.802429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:50.802390 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.802676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:50.802472 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:51.625517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.625475 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.625717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.625541 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:51.680887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.680852 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9"] Nov 03 16:39:51.681964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.681684 6001 cpu_manager.go:414] "containerMap: removing container" podUID="50d53fd1-efa7-4ca7-b113-d84d18e14038" containerName="revoke-keys" Nov 03 16:39:51.681964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.681703 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50d53fd1-efa7-4ca7-b113-d84d18e14038" containerName="revoke-keys" Nov 03 16:39:51.681964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.681894 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50d53fd1-efa7-4ca7-b113-d84d18e14038" containerName="revoke-keys" Nov 03 16:39:51.685920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.685901 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.701986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.701957 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:39:51.708270 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.708238 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9"] Nov 03 16:39:51.710203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.710180 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:39:51.710288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.710256 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:39:51.710378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.710364 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:39:51.741924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.741891 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.742123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.742079 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsgkb\" (UniqueName: \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.742266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.742247 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.801619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.801575 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.801806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.801660 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Nov 03 16:39:51.847496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.843655 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.847496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.843853 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsgkb\" (UniqueName: \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.847496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.844014 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.848379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.847944 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.874547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.872144 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:51.875351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:51.875317 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsgkb\" (UniqueName: \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\") pod \"maas-api-revoke-sub-406bd897-669f9\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:52.010665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.008693 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:52.340780 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:39:52.340677 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:39:52.402763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.385489 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9"] Nov 03 16:39:52.629529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.627911 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:52.629529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.627980 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:52.811949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.811917 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:52.812551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.812531 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:52.814488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.813686 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:39:52.888135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.888054 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f"] Nov 03 16:39:52.894597 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.894569 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:52.910251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:52.910215 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f"] Nov 03 16:39:53.021740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.021701 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.021954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.021818 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.021954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.021852 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f7v\" (UniqueName: \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.145524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.127558 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.145524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.127657 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.145524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.127698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f7v\" (UniqueName: \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.145524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.128635 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.145524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.131910 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.193847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.193774 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f7v\" (UniqueName: \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\") pod \"maas-api-revoke-sub-e469e595-b9d4f\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.219944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.219906 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:53.451763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.449983 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" event={"ID":"d335c03e-e302-4fba-b0dc-a00f46190420","Type":"ContainerStarted","Data":"d76c88d84eea130650e2efd51d23d1cfcbaf80ee3b07d33b679fef594dad6de3"} Nov 03 16:39:53.451763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.450048 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" event={"ID":"d335c03e-e302-4fba-b0dc-a00f46190420","Type":"ContainerStarted","Data":"c3dc7be77980c4b6023fb21804daabed4f73d8c64bf4e7ec8f3e119f68749661"} Nov 03 16:39:53.521790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.511158 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" podStartSLOduration=2.511136191 podStartE2EDuration="2.511136191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:53.500859199 +0000 UTC m=+1488.834072772" watchObservedRunningTime="2026-11-03 16:39:53.511136191 +0000 UTC m=+1488.844349762" Nov 03 16:39:53.666619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.666532 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:53.666842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:53.666671 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:54.044645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.044531 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f"] Nov 03 16:39:54.495471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.494804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" event={"ID":"8943f3b0-aaac-41d7-bedc-e4da89f59359","Type":"ContainerStarted","Data":"e25cf2297c6d9184db000127ed06b6ea6505eae10240d0652cf7e482fa7bc70a"} Nov 03 16:39:54.500478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.497177 6001 generic.go:356] "Generic (PLEG): container finished" podID="d335c03e-e302-4fba-b0dc-a00f46190420" containerID="d76c88d84eea130650e2efd51d23d1cfcbaf80ee3b07d33b679fef594dad6de3" exitCode=0 Nov 03 16:39:54.529470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.514744 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" event={"ID":"d335c03e-e302-4fba-b0dc-a00f46190420","Type":"ContainerDied","Data":"d76c88d84eea130650e2efd51d23d1cfcbaf80ee3b07d33b679fef594dad6de3"} Nov 03 16:39:54.660538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.657228 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:54.660538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.657795 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:54.869468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.869385 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:54.873499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:54.870905 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:55.002469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.002339 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.092905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.092855 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:55.244651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.243529 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.388489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.387556 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.530926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.530552 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.549562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.548697 6001 generic.go:356] "Generic (PLEG): container finished" podID="8943f3b0-aaac-41d7-bedc-e4da89f59359" containerID="449eb97a377deb4dc5208fbc300b81696a15c06b45a5eb0af2f94cfdb32e0d76" exitCode=0 Nov 03 16:39:55.549562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.549414 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" event={"ID":"8943f3b0-aaac-41d7-bedc-e4da89f59359","Type":"ContainerDied","Data":"449eb97a377deb4dc5208fbc300b81696a15c06b45a5eb0af2f94cfdb32e0d76"} Nov 03 16:39:55.651469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.633711 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:55.651469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.633764 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:55.763129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.763080 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.764493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:55.764032 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.098684 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.099674 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.122042 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.123793 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.123988 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:39:56.131475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.124141 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:39:56.272468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.247098 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.325280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.325250 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:39:56.394372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.394256 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.395140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.395112 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:39:56.756140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.756081 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.056910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.784261 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.056910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.056047 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:57.056910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:56.823424 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.100915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.100576 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.100915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.100820 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:39:57.169480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.126938 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:57.341530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.340705 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:39:57.376754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.365089 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:39:57.376754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.365165 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:57.423470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.422925 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:39:57.506822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.506768 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.647876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.647781 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.679750 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.651696 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 16:39:57.704952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.704917 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:57.967364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.967254 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\") pod \"d335c03e-e302-4fba-b0dc-a00f46190420\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " Nov 03 16:39:57.967704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:57.967680 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\") pod \"d335c03e-e302-4fba-b0dc-a00f46190420\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " Nov 03 16:39:58.013471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.013090 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca" pod "d335c03e-e302-4fba-b0dc-a00f46190420" (UID: "d335c03e-e302-4fba-b0dc-a00f46190420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:58.038477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.037546 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:58.038477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.037641 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.063802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.063749 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.064456 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.064404 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 16:39:58.068533 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.065166 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\" (UniqueName: \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\") pod \"d335c03e-e302-4fba-b0dc-a00f46190420\" (UID: \"d335c03e-e302-4fba-b0dc-a00f46190420\") " Nov 03 16:39:58.100472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.069578 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:58.312490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.128030 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.312490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.128098 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.312490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.174671 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.342117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.341488 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.376665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.376627 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" event={"ID":"d335c03e-e302-4fba-b0dc-a00f46190420","Type":"ContainerDied","Data":"c3dc7be77980c4b6023fb21804daabed4f73d8c64bf4e7ec8f3e119f68749661"} Nov 03 16:39:58.376909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.376888 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3dc7be77980c4b6023fb21804daabed4f73d8c64bf4e7ec8f3e119f68749661" Nov 03 16:39:58.377112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.377097 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9" Nov 03 16:39:58.379506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.379481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:39:58.425605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.425552 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:39:58.603374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.589716 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca" pod "d335c03e-e302-4fba-b0dc-a00f46190420" (UID: "d335c03e-e302-4fba-b0dc-a00f46190420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:58.603374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.589860 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb" pod "d335c03e-e302-4fba-b0dc-a00f46190420" (UID: "d335c03e-e302-4fba-b0dc-a00f46190420"). InnerVolumeSpecName "kube-api-access-dsgkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:58.654639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.654563 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsgkb\" (UniqueName: \"kubernetes.io/projected/d335c03e-e302-4fba-b0dc-a00f46190420-kube-api-access-dsgkb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:58.654884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:58.654863 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d335c03e-e302-4fba-b0dc-a00f46190420-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:59.131283 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.131244 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.132634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.131620 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:39:59.132634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.131702 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 16:39:59.229492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.220061 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:39:59.229492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.224767 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:59.241483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.230356 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:39:59.298626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.298583 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:39:59.301272 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.299751 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:39:59.357783 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.357747 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\" (UniqueName: \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\") pod \"8943f3b0-aaac-41d7-bedc-e4da89f59359\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " Nov 03 16:39:59.358585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.358549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\") pod \"8943f3b0-aaac-41d7-bedc-e4da89f59359\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " Nov 03 16:39:59.358867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.358850 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\") pod \"8943f3b0-aaac-41d7-bedc-e4da89f59359\" (UID: \"8943f3b0-aaac-41d7-bedc-e4da89f59359\") " Nov 03 16:39:59.360580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.360548 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.361162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.361135 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca" pod "8943f3b0-aaac-41d7-bedc-e4da89f59359" (UID: "8943f3b0-aaac-41d7-bedc-e4da89f59359"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:39:59.400520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.390489 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:39:59.405905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.405875 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:39:59.553470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.538193 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v" pod "8943f3b0-aaac-41d7-bedc-e4da89f59359" (UID: "8943f3b0-aaac-41d7-bedc-e4da89f59359"). InnerVolumeSpecName "kube-api-access-58f7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:39:59.572891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.564014 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:59.572891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.564052 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f7v\" (UniqueName: \"kubernetes.io/projected/8943f3b0-aaac-41d7-bedc-e4da89f59359-kube-api-access-58f7v\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:39:59.576402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.576359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" event={"ID":"8943f3b0-aaac-41d7-bedc-e4da89f59359","Type":"ContainerDied","Data":"e25cf2297c6d9184db000127ed06b6ea6505eae10240d0652cf7e482fa7bc70a"} Nov 03 16:39:59.576720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.576697 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e25cf2297c6d9184db000127ed06b6ea6505eae10240d0652cf7e482fa7bc70a" Nov 03 16:39:59.576936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.576920 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f" Nov 03 16:39:59.577569 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.577548 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca" pod "8943f3b0-aaac-41d7-bedc-e4da89f59359" (UID: "8943f3b0-aaac-41d7-bedc-e4da89f59359"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:39:59.628542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.628496 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:59.628711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.628566 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:59.665807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:39:59.665705 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8943f3b0-aaac-41d7-bedc-e4da89f59359-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:00.669650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:00.669608 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:00.670135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:00.669674 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:00.968133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:00.968030 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:40:00.968508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:00.968481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:40:00.969437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:00.969417 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:40:03.295487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.295434 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9"] Nov 03 16:40:03.297306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297275 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d335c03e-e302-4fba-b0dc-a00f46190420" containerName="revoke-keys" Nov 03 16:40:03.297430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297309 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d335c03e-e302-4fba-b0dc-a00f46190420" containerName="revoke-keys" Nov 03 16:40:03.297430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297395 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8943f3b0-aaac-41d7-bedc-e4da89f59359" containerName="revoke-keys" Nov 03 16:40:03.297430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297407 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8943f3b0-aaac-41d7-bedc-e4da89f59359" containerName="revoke-keys" Nov 03 16:40:03.297816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297795 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8943f3b0-aaac-41d7-bedc-e4da89f59359" containerName="revoke-keys" Nov 03 16:40:03.297886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.297847 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d335c03e-e302-4fba-b0dc-a00f46190420" containerName="revoke-keys" Nov 03 16:40:03.304341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.304315 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.320348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.320326 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9"] Nov 03 16:40:03.324203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.324180 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:03.324844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.324810 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.324944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.324862 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.325065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.325047 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4krj\" (UniqueName: \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.328913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.328887 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:40:03.329041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.328970 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:40:03.329104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.329061 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:40:03.426733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.426690 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4krj\" (UniqueName: \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.426913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.426831 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.426913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.426853 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.427514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.427490 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.429241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.429217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.468961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.468927 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4krj\" (UniqueName: \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\") pod \"maas-api-revoke-sub-43033229-sv4t9\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.624004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.623919 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:03.929439 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:03.929405 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371926e7_0985_4f2e_9acf_8eb84ea521ce.slice/crio-cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e WatchSource:0}: Error finding container cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e: Status 404 returned error can't find the container with id cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e Nov 03 16:40:03.930927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:03.930904 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9"] Nov 03 16:40:04.621897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:04.621805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" event={"ID":"371926e7-0985-4f2e-9acf-8eb84ea521ce","Type":"ContainerStarted","Data":"d3ff1073d0aaef100b191bb6a4e5369dbb611c87c5d72bbda65d741d99e26032"} Nov 03 16:40:04.621897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:04.621867 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" event={"ID":"371926e7-0985-4f2e-9acf-8eb84ea521ce","Type":"ContainerStarted","Data":"cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e"} Nov 03 16:40:04.666768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:04.666712 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" podStartSLOduration=1.666698209 podStartE2EDuration="1.666698209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:04.664751827 +0000 UTC m=+1499.997965403" watchObservedRunningTime="2026-11-03 16:40:04.666698209 +0000 UTC m=+1499.999911778" Nov 03 16:40:04.995689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:04.995661 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:40:05.007128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:05.007105 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:40:05.631776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:05.631742 6001 generic.go:356] "Generic (PLEG): container finished" podID="371926e7-0985-4f2e-9acf-8eb84ea521ce" containerID="d3ff1073d0aaef100b191bb6a4e5369dbb611c87c5d72bbda65d741d99e26032" exitCode=0 Nov 03 16:40:05.632250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:05.631828 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" event={"ID":"371926e7-0985-4f2e-9acf-8eb84ea521ce","Type":"ContainerDied","Data":"d3ff1073d0aaef100b191bb6a4e5369dbb611c87c5d72bbda65d741d99e26032"} Nov 03 16:40:07.149247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.149192 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.303476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.300566 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.429695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.428602 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": context deadline exceeded" Nov 03 16:40:07.960215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.944720 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:07.970532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.967184 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" event={"ID":"371926e7-0985-4f2e-9acf-8eb84ea521ce","Type":"ContainerDied","Data":"cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e"} Nov 03 16:40:07.970824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:07.970796 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc004bde5f1a07137e37fe94b85c33c850ff76ed385dd7d8166c36d3cb12f63e" Nov 03 16:40:08.037968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.037927 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\") pod \"371926e7-0985-4f2e-9acf-8eb84ea521ce\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " Nov 03 16:40:08.038324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.038298 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\") pod \"371926e7-0985-4f2e-9acf-8eb84ea521ce\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " Nov 03 16:40:08.063755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.063702 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca" pod "371926e7-0985-4f2e-9acf-8eb84ea521ce" (UID: "371926e7-0985-4f2e-9acf-8eb84ea521ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:08.090838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.090791 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.152613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.148864 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\" (UniqueName: \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\") pod \"371926e7-0985-4f2e-9acf-8eb84ea521ce\" (UID: \"371926e7-0985-4f2e-9acf-8eb84ea521ce\") " Nov 03 16:40:08.152613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.149561 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:08.153308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.153268 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.153877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.153845 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:40:08.158555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.158522 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.195520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.175590 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:08.279323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.201654 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj" pod "371926e7-0985-4f2e-9acf-8eb84ea521ce" (UID: "371926e7-0985-4f2e-9acf-8eb84ea521ce"). InnerVolumeSpecName "kube-api-access-g4krj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:08.376052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.355779 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca" pod "371926e7-0985-4f2e-9acf-8eb84ea521ce" (UID: "371926e7-0985-4f2e-9acf-8eb84ea521ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:08.379639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.355652 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:40:08.393472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.357240 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.409881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.355592 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.411663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.410954 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371926e7-0985-4f2e-9acf-8eb84ea521ce-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:08.411663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.410988 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4krj\" (UniqueName: \"kubernetes.io/projected/371926e7-0985-4f2e-9acf-8eb84ea521ce-kube-api-access-g4krj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:08.412064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.412046 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:08.960868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:08.959677 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": context deadline exceeded" Nov 03 16:40:09.003895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.002599 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.003895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.002672 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.015653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.015616 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9" Nov 03 16:40:09.109470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.106293 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.140513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.127308 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:09.157549 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.157509 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.200662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.200613 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.247512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.234756 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:09.445483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.436275 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:40:09.492387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.492338 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:40:09.504540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.493041 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:09.510848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.510808 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.511141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.511106 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.540880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.540834 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.541628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.541603 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:09.639663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.639575 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.646625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:09.646481 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.138478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.120563 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:10.159839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.159798 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.160616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.151653 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.260463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.244278 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:40:10.260463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.248210 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:40:10.266945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.265113 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:10.266945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.265272 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:10.339961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.339931 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:10.370334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.365523 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:10.381619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.381584 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:10.395789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.395631 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:10.400694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.400665 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:10.427013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.426983 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:10.441600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:10.441562 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:13.003719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.003685 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4"] Nov 03 16:40:13.004375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.004358 6001 cpu_manager.go:414] "containerMap: removing container" podUID="371926e7-0985-4f2e-9acf-8eb84ea521ce" containerName="revoke-keys" Nov 03 16:40:13.004439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.004376 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371926e7-0985-4f2e-9acf-8eb84ea521ce" containerName="revoke-keys" Nov 03 16:40:13.004519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.004509 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="371926e7-0985-4f2e-9acf-8eb84ea521ce" containerName="revoke-keys" Nov 03 16:40:13.008379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.008361 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.028028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.027998 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4"] Nov 03 16:40:13.035935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.035905 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:40:13.035935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.035915 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:40:13.039655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.039630 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:13.039792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.039679 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:40:13.170017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.169932 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.170197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.170129 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqlk\" (UniqueName: \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.170197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.170178 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.271519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.271479 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.271723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.271705 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.271860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.271844 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqlk\" (UniqueName: \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.272202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.272173 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.274887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.274862 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.296751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.296718 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqlk\" (UniqueName: \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\") pod \"maas-api-revoke-sub-5ba401d0-jf9v4\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.322551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.322523 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:13.638534 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:13.638504 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4d706a_281c_4f9e_9faf_081b4e8252e6.slice/crio-0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e WatchSource:0}: Error finding container 0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e: Status 404 returned error can't find the container with id 0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e Nov 03 16:40:13.639929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:13.639903 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4"] Nov 03 16:40:14.316267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:14.316229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" event={"ID":"ad4d706a-281c-4f9e-9faf-081b4e8252e6","Type":"ContainerStarted","Data":"9dbf7863d91193dd6a3130f528f84a08622877629ffcd42d8baf51dacce7b40d"} Nov 03 16:40:14.316729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:14.316285 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" event={"ID":"ad4d706a-281c-4f9e-9faf-081b4e8252e6","Type":"ContainerStarted","Data":"0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e"} Nov 03 16:40:14.366040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:14.365988 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" podStartSLOduration=2.365974039 podStartE2EDuration="2.365974039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:14.362605628 +0000 UTC m=+1509.695819215" watchObservedRunningTime="2026-11-03 16:40:14.365974039 +0000 UTC m=+1509.699187630" Nov 03 16:40:15.326665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:15.326626 6001 generic.go:356] "Generic (PLEG): container finished" podID="ad4d706a-281c-4f9e-9faf-081b4e8252e6" containerID="9dbf7863d91193dd6a3130f528f84a08622877629ffcd42d8baf51dacce7b40d" exitCode=0 Nov 03 16:40:15.327161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:15.326714 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" event={"ID":"ad4d706a-281c-4f9e-9faf-081b4e8252e6","Type":"ContainerDied","Data":"9dbf7863d91193dd6a3130f528f84a08622877629ffcd42d8baf51dacce7b40d"} Nov 03 16:40:16.499314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.499284 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:16.619125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.619025 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\") pod \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " Nov 03 16:40:16.619348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.619325 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\" (UniqueName: \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\") pod \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " Nov 03 16:40:16.619424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.619374 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\") pod \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\" (UID: \"ad4d706a-281c-4f9e-9faf-081b4e8252e6\") " Nov 03 16:40:16.619424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.619389 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca" pod "ad4d706a-281c-4f9e-9faf-081b4e8252e6" (UID: "ad4d706a-281c-4f9e-9faf-081b4e8252e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:16.619956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.619934 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:16.621505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.621479 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca" pod "ad4d706a-281c-4f9e-9faf-081b4e8252e6" (UID: "ad4d706a-281c-4f9e-9faf-081b4e8252e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:16.621659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.621636 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk" pod "ad4d706a-281c-4f9e-9faf-081b4e8252e6" (UID: "ad4d706a-281c-4f9e-9faf-081b4e8252e6"). InnerVolumeSpecName "kube-api-access-nxqlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:16.721049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.721014 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqlk\" (UniqueName: \"kubernetes.io/projected/ad4d706a-281c-4f9e-9faf-081b4e8252e6-kube-api-access-nxqlk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:16.721049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:16.721049 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad4d706a-281c-4f9e-9faf-081b4e8252e6-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:17.358591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:17.358551 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" Nov 03 16:40:17.358771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:17.358587 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4" event={"ID":"ad4d706a-281c-4f9e-9faf-081b4e8252e6","Type":"ContainerDied","Data":"0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e"} Nov 03 16:40:17.358771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:17.358673 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cad8ded7c4012c79011c590a81b971a711bcc88d3e37ea360efc403390ac81e" Nov 03 16:40:21.370602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.370541 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:21.441464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.441411 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:40:21.462469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.442386 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ad4d706a-281c-4f9e-9faf-081b4e8252e6" containerName="revoke-keys" Nov 03 16:40:21.462469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.442411 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4d706a-281c-4f9e-9faf-081b4e8252e6" containerName="revoke-keys" Nov 03 16:40:21.462469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.442640 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4d706a-281c-4f9e-9faf-081b4e8252e6" containerName="revoke-keys" Nov 03 16:40:21.462469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.458631 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.543230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.543195 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:40:21.738779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.738738 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.739049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.739025 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.739220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.739199 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc69x\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.739355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.739337 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.739522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.739505 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.762771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.762726 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.763061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.763036 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.763253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.763234 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.763462 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.763425 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903094 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903149 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903194 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903257 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903334 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903530 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903636 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc69x\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.903670 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.908469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.904676 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.909104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.909017 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.917519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.909965 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.917519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.910914 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.917519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.911175 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:21.948121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.948038 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:22.021470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:21.976490 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:22.125469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.119848 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc69x\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:22.168955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.168905 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.192681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.192643 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:22.265470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.250085 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.300470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.299966 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:22.378539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.353590 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.378539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.374132 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.378539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.374540 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:22.388897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.388528 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.414471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.412878 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.506845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:22.506794 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.157685 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:40:23.157643 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:40:23.173594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.173542 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.175873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.175843 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:23.271505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.269118 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.271505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.269944 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:23.382476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.356529 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.382476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.357325 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:40:23.382476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.379782 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.425576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.398517 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.425576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.398950 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:23.454467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.450554 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.478503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.478462 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:23.544803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.507581 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.545506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.545478 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:23.956155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.956104 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:23.956523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:23.956480 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.008098 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.008067 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:24.031285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.031238 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.109624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.109578 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:24.109934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.109891 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.184469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.175539 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:40:24.239112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.233848 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:24.278437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.277802 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.359386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.356045 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k"] Nov 03 16:40:24.394617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.393963 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.403536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.400540 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.403536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.400837 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.421644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.419880 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:40:24.421644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.420046 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:24.421644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.420175 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:40:24.421644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.420257 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:40:24.462975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.462275 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.549000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.544668 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.549000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.544755 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.549000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.544896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chrfx\" (UniqueName: \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.549000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.545160 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.585549 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.585104 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:24.614576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.612537 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:24.614576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.612597 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.625478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.617116 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:40:24.638733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.637716 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:24.645550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.643162 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:24.647716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.646325 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.647716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.646396 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.647716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.646653 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chrfx\" (UniqueName: \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.648972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.648948 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.662190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.662159 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.690872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.690818 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k"] Nov 03 16:40:24.691704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.691677 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chrfx\" (UniqueName: \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\") pod \"maas-api-revoke-sub-b4738d5f-2bc8k\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:24.702384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.702357 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:24.703920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.703887 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:24.704049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.703967 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:24.704049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.704012 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:24.729726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.729693 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:40:24.737534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:24.734366 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:25.035960 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.035923 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l"] Nov 03 16:40:25.041178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.041153 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.062503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.059981 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsvt\" (UniqueName: \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.062503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.060159 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.062503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.060244 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.062503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.061235 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l"] Nov 03 16:40:25.087773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.083258 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k"] Nov 03 16:40:25.136782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.136746 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" event={"ID":"99a4d895-7d0a-4daa-acfb-7da07325fc73","Type":"ContainerStarted","Data":"3cdf30b9a26c3265e2689cce8a58563e5b55d951a1321ec3fff6e7367c87e332"} Nov 03 16:40:25.138744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.138711 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" event={"ID":"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6","Type":"ContainerStarted","Data":"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046"} Nov 03 16:40:25.138912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.138771 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" event={"ID":"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6","Type":"ContainerStarted","Data":"89489e489bae75bb1a82f816c373a285c7239e6fe5c23a3d7867e818fc8f3ec2"} Nov 03 16:40:25.161913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.161881 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsvt\" (UniqueName: \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.162130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.161990 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.162130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.162059 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.162888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.162862 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.187818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.187750 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podStartSLOduration=4.187730744 podStartE2EDuration="4.187730744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:25.185204206 +0000 UTC m=+1520.518417777" watchObservedRunningTime="2026-11-03 16:40:25.187730744 +0000 UTC m=+1520.520944314" Nov 03 16:40:25.245170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.245122 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.245335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.245226 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsvt\" (UniqueName: \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\") pod \"maas-api-revoke-sub-13f0e46b-tbm7l\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.311739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.311709 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:25.313325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.313302 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:25.313399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.313351 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:25.358179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.358095 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:25.683627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:25.683587 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l"] Nov 03 16:40:26.174023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.173983 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" event={"ID":"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd","Type":"ContainerStarted","Data":"1468442e2b51c8d5e2590d9a2c3060b205a28ffe00c091689a9934b4f35cf582"} Nov 03 16:40:26.175992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.175959 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" event={"ID":"99a4d895-7d0a-4daa-acfb-7da07325fc73","Type":"ContainerStarted","Data":"224e11c854e7a83d6c756e2fc66179f8fa953471c17fa69554dae0f81666f439"} Nov 03 16:40:26.232236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.232080 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" podStartSLOduration=3.2320647 podStartE2EDuration="3.2320647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:26.228742286 +0000 UTC m=+1521.561955854" watchObservedRunningTime="2026-11-03 16:40:26.2320647 +0000 UTC m=+1521.565278268" Nov 03 16:40:26.276235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.276191 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq"] Nov 03 16:40:26.281999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.281970 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.306614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.306576 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq"] Nov 03 16:40:26.313064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.313031 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:26.313238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.313086 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:26.407843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.407806 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.408051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.408033 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.408162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.408148 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqfv\" (UniqueName: \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.509158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.509061 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.509313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.509219 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.509313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.509289 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqfv\" (UniqueName: \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.509942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.509905 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.511491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.511435 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.534257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.534226 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqfv\" (UniqueName: \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\") pod \"maas-api-revoke-sub-70bd2474-vrxdq\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.636052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.636014 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:26.982697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:26.982663 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq"] Nov 03 16:40:27.185616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.185580 6001 generic.go:356] "Generic (PLEG): container finished" podID="1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" containerID="5e2fc06388743325db6dc6df05728548aa5171f68a3b71490086c20361356c49" exitCode=0 Nov 03 16:40:27.185775 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.185663 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" event={"ID":"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd","Type":"ContainerDied","Data":"5e2fc06388743325db6dc6df05728548aa5171f68a3b71490086c20361356c49"} Nov 03 16:40:27.187453 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.187413 6001 generic.go:356] "Generic (PLEG): container finished" podID="99a4d895-7d0a-4daa-acfb-7da07325fc73" containerID="224e11c854e7a83d6c756e2fc66179f8fa953471c17fa69554dae0f81666f439" exitCode=0 Nov 03 16:40:27.187587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.187472 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" event={"ID":"99a4d895-7d0a-4daa-acfb-7da07325fc73","Type":"ContainerDied","Data":"224e11c854e7a83d6c756e2fc66179f8fa953471c17fa69554dae0f81666f439"} Nov 03 16:40:27.189280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.189253 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" event={"ID":"513f04c2-e3bf-485c-8597-f30be2b69ddd","Type":"ContainerStarted","Data":"ce62e490de9441e4b01ae1dfa3708b6e2b917597f794170e85600bcc19c70198"} Nov 03 16:40:27.311978 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.311935 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:27.312158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:27.311998 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:28.202250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.202212 6001 generic.go:356] "Generic (PLEG): container finished" podID="513f04c2-e3bf-485c-8597-f30be2b69ddd" containerID="bbf483a18730f7b1b4818b034520b9dac1607e6bf563b20bc416521e26eb4f74" exitCode=0 Nov 03 16:40:28.202684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.202301 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" event={"ID":"513f04c2-e3bf-485c-8597-f30be2b69ddd","Type":"ContainerDied","Data":"bbf483a18730f7b1b4818b034520b9dac1607e6bf563b20bc416521e26eb4f74"} Nov 03 16:40:28.312841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.312006 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:28.312841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.312058 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:28.429579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.429407 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:28.478559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.478528 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:28.539991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.539955 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\") pod \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " Nov 03 16:40:28.540288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540263 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\") pod \"99a4d895-7d0a-4daa-acfb-7da07325fc73\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " Nov 03 16:40:28.540725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540521 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca" pod "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" (UID: "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:28.540725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540584 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\") pod \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " Nov 03 16:40:28.540880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540765 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\" (UniqueName: \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\") pod \"99a4d895-7d0a-4daa-acfb-7da07325fc73\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " Nov 03 16:40:28.540880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540873 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\") pod \"99a4d895-7d0a-4daa-acfb-7da07325fc73\" (UID: \"99a4d895-7d0a-4daa-acfb-7da07325fc73\") " Nov 03 16:40:28.541260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.540948 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\" (UniqueName: \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\") pod \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\" (UID: \"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd\") " Nov 03 16:40:28.542069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.542043 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca" pod "99a4d895-7d0a-4daa-acfb-7da07325fc73" (UID: "99a4d895-7d0a-4daa-acfb-7da07325fc73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:28.572200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.568733 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:28.572200 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.568781 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:28.573483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.573417 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca" pod "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" (UID: "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:28.574519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.574486 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx" pod "99a4d895-7d0a-4daa-acfb-7da07325fc73" (UID: "99a4d895-7d0a-4daa-acfb-7da07325fc73"). InnerVolumeSpecName "kube-api-access-chrfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:28.574621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.574535 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca" pod "99a4d895-7d0a-4daa-acfb-7da07325fc73" (UID: "99a4d895-7d0a-4daa-acfb-7da07325fc73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:28.575340 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.575314 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt" pod "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" (UID: "1ed08ab7-aae5-4ae0-87cd-da29b3024fcd"). InnerVolumeSpecName "kube-api-access-6rsvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:28.670403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.670299 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99a4d895-7d0a-4daa-acfb-7da07325fc73-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:28.670685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.670666 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:28.670791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.670778 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chrfx\" (UniqueName: \"kubernetes.io/projected/99a4d895-7d0a-4daa-acfb-7da07325fc73-kube-api-access-chrfx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:28.670878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:28.670866 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsvt\" (UniqueName: \"kubernetes.io/projected/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd-kube-api-access-6rsvt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.213564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.213531 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" Nov 03 16:40:29.214087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.213530 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l" event={"ID":"1ed08ab7-aae5-4ae0-87cd-da29b3024fcd","Type":"ContainerDied","Data":"1468442e2b51c8d5e2590d9a2c3060b205a28ffe00c091689a9934b4f35cf582"} Nov 03 16:40:29.214087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.213664 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1468442e2b51c8d5e2590d9a2c3060b205a28ffe00c091689a9934b4f35cf582" Nov 03 16:40:29.215394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.215360 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" event={"ID":"99a4d895-7d0a-4daa-acfb-7da07325fc73","Type":"ContainerDied","Data":"3cdf30b9a26c3265e2689cce8a58563e5b55d951a1321ec3fff6e7367c87e332"} Nov 03 16:40:29.215533 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.215419 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cdf30b9a26c3265e2689cce8a58563e5b55d951a1321ec3fff6e7367c87e332" Nov 03 16:40:29.215533 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.215431 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k" Nov 03 16:40:29.314233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.314193 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:29.314395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.314262 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:29.439054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.439025 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:29.597192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.597134 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\") pod \"513f04c2-e3bf-485c-8597-f30be2b69ddd\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " Nov 03 16:40:29.597394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.597265 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\") pod \"513f04c2-e3bf-485c-8597-f30be2b69ddd\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " Nov 03 16:40:29.597394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.597359 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\" (UniqueName: \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\") pod \"513f04c2-e3bf-485c-8597-f30be2b69ddd\" (UID: \"513f04c2-e3bf-485c-8597-f30be2b69ddd\") " Nov 03 16:40:29.597930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.597903 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca" pod "513f04c2-e3bf-485c-8597-f30be2b69ddd" (UID: "513f04c2-e3bf-485c-8597-f30be2b69ddd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:29.598242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.598221 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.599764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.599740 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv" pod "513f04c2-e3bf-485c-8597-f30be2b69ddd" (UID: "513f04c2-e3bf-485c-8597-f30be2b69ddd"). InnerVolumeSpecName "kube-api-access-mcqfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:29.600048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.600027 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca" pod "513f04c2-e3bf-485c-8597-f30be2b69ddd" (UID: "513f04c2-e3bf-485c-8597-f30be2b69ddd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:29.639210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639172 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:40:29.639876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639859 6001 cpu_manager.go:414] "containerMap: removing container" podUID="99a4d895-7d0a-4daa-acfb-7da07325fc73" containerName="revoke-keys" Nov 03 16:40:29.639924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639878 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a4d895-7d0a-4daa-acfb-7da07325fc73" containerName="revoke-keys" Nov 03 16:40:29.639924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639890 6001 cpu_manager.go:414] "containerMap: removing container" podUID="513f04c2-e3bf-485c-8597-f30be2b69ddd" containerName="revoke-keys" Nov 03 16:40:29.639924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639896 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513f04c2-e3bf-485c-8597-f30be2b69ddd" containerName="revoke-keys" Nov 03 16:40:29.640020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639928 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" containerName="revoke-keys" Nov 03 16:40:29.640020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.639934 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" containerName="revoke-keys" Nov 03 16:40:29.640084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.640034 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a4d895-7d0a-4daa-acfb-7da07325fc73" containerName="revoke-keys" Nov 03 16:40:29.640084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.640044 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="513f04c2-e3bf-485c-8597-f30be2b69ddd" containerName="revoke-keys" Nov 03 16:40:29.640084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.640056 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" containerName="revoke-keys" Nov 03 16:40:29.650539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.649380 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.699268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.699239 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:40:29.700125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.700098 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqfv\" (UniqueName: \"kubernetes.io/projected/513f04c2-e3bf-485c-8597-f30be2b69ddd-kube-api-access-mcqfv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.700125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.700126 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/513f04c2-e3bf-485c-8597-f30be2b69ddd-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:29.801147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801101 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801231 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws7p2\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801390 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801423 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801543 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801597 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801583 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801613 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801707 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.801761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.801749 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903406 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903481 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903519 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903795 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.903992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.903973 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904004 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904041 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904111 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904147 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904209 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904224 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws7p2\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904562 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.904782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.904762 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.906213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.906195 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.906264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.906208 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.927700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.927672 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws7p2\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:29.927883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:29.927861 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:30.005228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.005193 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:30.238214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.238180 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" Nov 03 16:40:30.238774 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.238197 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq" event={"ID":"513f04c2-e3bf-485c-8597-f30be2b69ddd","Type":"ContainerDied","Data":"ce62e490de9441e4b01ae1dfa3708b6e2b917597f794170e85600bcc19c70198"} Nov 03 16:40:30.238956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.238936 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce62e490de9441e4b01ae1dfa3708b6e2b917597f794170e85600bcc19c70198" Nov 03 16:40:30.312363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.312316 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:30.312574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.312391 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:30.373637 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:30.373598 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e653437_91ef_4385_a9fc_4c763318a4a5.slice/crio-bd1e3bebadd3ae9460dafd729675d6cef9b17f089ca38681160dfe256642c8be WatchSource:0}: Error finding container bd1e3bebadd3ae9460dafd729675d6cef9b17f089ca38681160dfe256642c8be: Status 404 returned error can't find the container with id bd1e3bebadd3ae9460dafd729675d6cef9b17f089ca38681160dfe256642c8be Nov 03 16:40:30.375629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.375596 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:40:30.376878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.376848 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:30.376933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.376915 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:30.376965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:30.376945 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:31.269480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:31.269106 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" event={"ID":"5e653437-91ef-4385-a9fc-4c763318a4a5","Type":"ContainerStarted","Data":"557eaf993f5cdf0609790ce0ebc1faf12fcc8e97f72ee67e948b771cd5920031"} Nov 03 16:40:31.269480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:31.269176 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" event={"ID":"5e653437-91ef-4385-a9fc-4c763318a4a5","Type":"ContainerStarted","Data":"bd1e3bebadd3ae9460dafd729675d6cef9b17f089ca38681160dfe256642c8be"} Nov 03 16:40:31.313068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:31.313032 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.313255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:31.313089 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Nov 03 16:40:31.333126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:31.333074 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podStartSLOduration=2.333058546 podStartE2EDuration="2.333058546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.330119732 +0000 UTC m=+1526.663333302" watchObservedRunningTime="2026-11-03 16:40:31.333058546 +0000 UTC m=+1526.666272172" Nov 03 16:40:32.006421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.006383 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:32.008362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.008333 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:32.008542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.008383 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:32.677472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.675843 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:32.679769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.679738 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:32.683473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:32.680929 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:40:33.006400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:33.006369 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:33.006619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:33.006426 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:34.006672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:34.006630 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:34.007075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:34.006697 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:35.006017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:35.005982 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:35.006206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:35.006038 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:36.006184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.006142 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:36.006643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.006234 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:36.442277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.442188 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:40:36.448586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.448558 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.481607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.481537 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:40:36.527036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.526994 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527101 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527163 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkk2\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527283 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527339 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527488 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527609 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527632 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.527779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.527754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.628718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.628685 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.628936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.628762 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.628936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.628870 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.628936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.628908 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.629115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.628962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.629115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629019 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629643 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629714 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkk2\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629745 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629850 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.629874 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.630254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.630631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.630625 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.631066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.630761 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.634224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.634197 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.635101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.635073 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.653897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.653865 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.658165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.658129 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkk2\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.704085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.703996 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw"] Nov 03 16:40:36.709786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.709755 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.727066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.727033 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:40:36.727884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.727861 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:40:36.728031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.727943 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:40:36.728031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.728016 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:40:36.732528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.732503 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw"] Nov 03 16:40:36.766895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.766860 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:36.833006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.832968 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.833187 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.833147 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95f7\" (UniqueName: \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.833187 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.833180 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.935252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.935214 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.935475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.935435 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95f7\" (UniqueName: \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.935553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.935500 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.936774 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.936741 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.938055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.937996 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:36.969272 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:36.969187 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95f7\" (UniqueName: \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\") pod \"maas-api-revoke-sub-37394ffa-m2ttw\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:37.006624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.006568 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:40:37.007027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.006650 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Nov 03 16:40:37.031740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.031695 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:37.171729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.171236 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:40:37.209502 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:37.207794 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510901d2_e32c_4348_b5d3_d6549bf76458.slice/crio-37f7ca8bff0ea720f727f7463cce2e2fe88f5a874ace8be29e47f88e0a000097 WatchSource:0}: Error finding container 37f7ca8bff0ea720f727f7463cce2e2fe88f5a874ace8be29e47f88e0a000097: Status 404 returned error can't find the container with id 37f7ca8bff0ea720f727f7463cce2e2fe88f5a874ace8be29e47f88e0a000097 Nov 03 16:40:37.213900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.213089 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:37.213900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.213187 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:37.213900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.213231 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:37.337300 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.335131 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" event={"ID":"510901d2-e32c-4348-b5d3-d6549bf76458","Type":"ContainerStarted","Data":"37f7ca8bff0ea720f727f7463cce2e2fe88f5a874ace8be29e47f88e0a000097"} Nov 03 16:40:37.396592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.396489 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podStartSLOduration=1.396466531 podStartE2EDuration="1.396466531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:37.391066045 +0000 UTC m=+1532.724279615" watchObservedRunningTime="2026-11-03 16:40:37.396466531 +0000 UTC m=+1532.729680105" Nov 03 16:40:37.501380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.501340 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw"] Nov 03 16:40:37.504686 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:37.504653 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c6e267_528c_43cc_9a7f_ad0e543c4b4d.slice/crio-8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7 WatchSource:0}: Error finding container 8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7: Status 404 returned error can't find the container with id 8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7 Nov 03 16:40:37.768068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:37.767973 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:38.097939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.097910 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:38.098525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.098344 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:38.099827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.099805 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:40:38.375385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.375292 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" event={"ID":"510901d2-e32c-4348-b5d3-d6549bf76458","Type":"ContainerStarted","Data":"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6"} Nov 03 16:40:38.378772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.378425 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" event={"ID":"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d","Type":"ContainerStarted","Data":"a315792ee2d90441a66803a2263eac2a1bbc27e07dbb2227287928ed1483a3e5"} Nov 03 16:40:38.378943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.378804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" event={"ID":"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d","Type":"ContainerStarted","Data":"8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7"} Nov 03 16:40:38.438595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.438532 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" podStartSLOduration=2.438518102 podStartE2EDuration="2.438518102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:38.436077079 +0000 UTC m=+1533.769290649" watchObservedRunningTime="2026-11-03 16:40:38.438518102 +0000 UTC m=+1533.771731671" Nov 03 16:40:38.794622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.794581 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:38.794929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:38.794896 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:39.388619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.388573 6001 generic.go:356] "Generic (PLEG): container finished" podID="d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" containerID="a315792ee2d90441a66803a2263eac2a1bbc27e07dbb2227287928ed1483a3e5" exitCode=0 Nov 03 16:40:39.389058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.388642 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" event={"ID":"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d","Type":"ContainerDied","Data":"a315792ee2d90441a66803a2263eac2a1bbc27e07dbb2227287928ed1483a3e5"} Nov 03 16:40:39.748469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.743898 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f"] Nov 03 16:40:39.770889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.770849 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:39.771117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.770912 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:39.771686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.771661 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:39.779230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.779200 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f"] Nov 03 16:40:39.943240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.943200 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:39.943504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.943390 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjz4m\" (UniqueName: \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:39.943579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:39.943551 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.065528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.049600 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjz4m\" (UniqueName: \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.065528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.049797 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.065528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.049959 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.065528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.062763 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.072390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.072356 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.086658 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.086628 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjz4m\" (UniqueName: \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\") pod \"maas-api-revoke-sub-77fc0ec4-z6c6f\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.090494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.090464 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:40.417322 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:40.417276 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef09499b_da21_4cf2_8d5b_7d4fb1325785.slice/crio-4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697 WatchSource:0}: Error finding container 4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697: Status 404 returned error can't find the container with id 4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697 Nov 03 16:40:40.420697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.420666 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f"] Nov 03 16:40:40.642354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.642324 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:40.685489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.685431 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\" (UniqueName: \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\") pod \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " Nov 03 16:40:40.685592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.685554 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\") pod \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " Nov 03 16:40:40.685661 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.685650 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\") pod \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\" (UID: \"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d\") " Nov 03 16:40:40.686017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.685978 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca" pod "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" (UID: "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:40.686208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.686186 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:40.687994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.687937 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca" pod "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" (UID: "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:40.688175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.688036 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7" pod "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" (UID: "d6c6e267-528c-43cc-9a7f-ad0e543c4b4d"). InnerVolumeSpecName "kube-api-access-v95f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:40.775530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.773612 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:40.775530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.773691 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:40.801746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.801702 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:40.801746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:40.801736 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95f7\" (UniqueName: \"kubernetes.io/projected/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d-kube-api-access-v95f7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:41.407914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.407799 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" event={"ID":"ef09499b-da21-4cf2-8d5b-7d4fb1325785","Type":"ContainerStarted","Data":"003b55d780fabaed1c3509bbeb368f1c10b76517ac914bd968dff9174c4cda4e"} Nov 03 16:40:41.408095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.407927 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" event={"ID":"ef09499b-da21-4cf2-8d5b-7d4fb1325785","Type":"ContainerStarted","Data":"4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697"} Nov 03 16:40:41.409501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.409476 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" Nov 03 16:40:41.409630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.409495 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw" event={"ID":"d6c6e267-528c-43cc-9a7f-ad0e543c4b4d","Type":"ContainerDied","Data":"8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7"} Nov 03 16:40:41.409630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.409540 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2becd8600b34b10baa8f41c761b7d6c18aae851ab23bd352984d1aa06f16f7" Nov 03 16:40:41.718507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.718391 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" podStartSLOduration=2.7183761779999998 podStartE2EDuration="2.718376178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:41.458858469 +0000 UTC m=+1536.792072038" watchObservedRunningTime="2026-11-03 16:40:41.718376178 +0000 UTC m=+1537.051589784" Nov 03 16:40:41.768937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.768895 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:41.769123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:41.768953 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:42.421071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:42.421038 6001 generic.go:356] "Generic (PLEG): container finished" podID="ef09499b-da21-4cf2-8d5b-7d4fb1325785" containerID="003b55d780fabaed1c3509bbeb368f1c10b76517ac914bd968dff9174c4cda4e" exitCode=0 Nov 03 16:40:42.421262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:42.421137 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" event={"ID":"ef09499b-da21-4cf2-8d5b-7d4fb1325785","Type":"ContainerDied","Data":"003b55d780fabaed1c3509bbeb368f1c10b76517ac914bd968dff9174c4cda4e"} Nov 03 16:40:42.771469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:42.770866 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:42.771469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:42.770934 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:43.770624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:43.767308 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:43.770624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:43.767375 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:44.138728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.103613 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.138728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.134882 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.459522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.442724 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.459994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.455562 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:40:44.470596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.470539 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.618471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.601786 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.655179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.655128 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:44.812171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.811999 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:44.812171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.812072 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:44.848029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:44.826474 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.135683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.135568 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.472684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.467760 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.472684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.468163 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 16:40:45.472684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.470937 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.485478 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.502846 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.503869 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ef09499b-da21-4cf2-8d5b-7d4fb1325785" containerName="revoke-keys" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.503889 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef09499b-da21-4cf2-8d5b-7d4fb1325785" containerName="revoke-keys" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.503930 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" containerName="revoke-keys" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.503938 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" containerName="revoke-keys" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.504088 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef09499b-da21-4cf2-8d5b-7d4fb1325785" containerName="revoke-keys" Nov 03 16:40:45.522784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.504109 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" containerName="revoke-keys" Nov 03 16:40:45.563634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.563576 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" podUID="045a4032-eeeb-46f7-a7f5-ebbbdc01558a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.26:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.564028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.563983 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.565798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.565736 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:45.566008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.565948 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:45.566186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.566149 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:45.566369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.566330 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:45.579487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.575151 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:45.605478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.603868 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 16:40:45.605478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.604274 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:45.656479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.645791 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:40:45.895467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.667415 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:45.997976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.685640 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\" (UniqueName: \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\") pod \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " Nov 03 16:40:46.002483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.998545 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\") pod \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " Nov 03 16:40:46.003596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.003571 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\") pod \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\" (UID: \"ef09499b-da21-4cf2-8d5b-7d4fb1325785\") " Nov 03 16:40:46.019131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.778104 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m" pod "ef09499b-da21-4cf2-8d5b-7d4fb1325785" (UID: "ef09499b-da21-4cf2-8d5b-7d4fb1325785"). InnerVolumeSpecName "kube-api-access-tjz4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:40:46.026547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:45.781580 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Nov 03 16:40:46.134299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.132829 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Nov 03 16:40:46.134299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.098752 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:46.134299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.111342 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:46.139210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.098653 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.154959 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.154918 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.191620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.191581 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.195701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.195642 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pvj\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.196281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.196260 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.205662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.205626 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.267724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.267690 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.268017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.267999 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.268295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.268278 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.268801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.268776 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjz4m\" (UniqueName: \"kubernetes.io/projected/ef09499b-da21-4cf2-8d5b-7d4fb1325785-kube-api-access-tjz4m\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:46.368824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.368723 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:40:46.369428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.027466 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca" pod "ef09499b-da21-4cf2-8d5b-7d4fb1325785" (UID: "ef09499b-da21-4cf2-8d5b-7d4fb1325785"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:40:46.483101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.475105 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca" pod "ef09499b-da21-4cf2-8d5b-7d4fb1325785" (UID: "ef09499b-da21-4cf2-8d5b-7d4fb1325785"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:40:46.483101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.476092 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:40:46.492758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.492727 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:40:46.493188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.493163 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:40:46.493342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.493241 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:40:46.493342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.493304 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:40:46.502388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.502356 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.525948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.525916 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.526225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.526202 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.527832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.526391 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.528182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.528161 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.528419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.528401 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.528611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.528588 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.534015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.533982 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.534286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.534266 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pvj\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.534632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.534613 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:46.534838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.534820 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef09499b-da21-4cf2-8d5b-7d4fb1325785-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:40:46.534931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.528907 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.535029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.534675 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.535666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.535642 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.542798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.542737 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.542798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.542747 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.544855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.544829 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.554469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.553734 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.560328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.560203 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.560979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.560943 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" Nov 03 16:40:46.561680 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.561486 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f" event={"ID":"ef09499b-da21-4cf2-8d5b-7d4fb1325785","Type":"ContainerDied","Data":"4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697"} Nov 03 16:40:46.561680 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.561534 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ee232a2a54986ea99e2b09a3c9d66748c4c41d0bfcb4dc1e958eea3dac37697" Nov 03 16:40:46.590471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.590387 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pvj\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:46.851648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:46.850363 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:47.026624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.026587 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:47.027742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.027711 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:47.029863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.029803 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:40:47.275112 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:47.275077 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e4fc6a_ca79_4856_bfa8_79a50eb434d9.slice/crio-b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af WatchSource:0}: Error finding container b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af: Status 404 returned error can't find the container with id b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af Nov 03 16:40:47.278124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.278091 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:47.278246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.278161 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:47.278246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.278188 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:40:47.279028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.278736 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:40:47.563490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.563336 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" event={"ID":"61e4fc6a-ca79-4856-bfa8-79a50eb434d9","Type":"ContainerStarted","Data":"62fe059155c73051583e03148466b6d7c42acce84957bc433d8adf24e7e55e48"} Nov 03 16:40:47.563934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.563494 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" event={"ID":"61e4fc6a-ca79-4856-bfa8-79a50eb434d9","Type":"ContainerStarted","Data":"b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af"} Nov 03 16:40:47.615429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.615358 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podStartSLOduration=3.615339758 podStartE2EDuration="3.615339758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:47.613318491 +0000 UTC m=+1542.946532061" watchObservedRunningTime="2026-11-03 16:40:47.615339758 +0000 UTC m=+1542.948553333" Nov 03 16:40:47.859803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:47.859606 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:48.869715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:48.869676 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:48.870314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:48.870273 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:49.854976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:49.854929 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:49.855290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:49.855255 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:50.916466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:50.916118 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:50.916466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:50.916233 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:51.867469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:51.866995 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:51.867469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:51.867080 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:52.272461 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:40:52.272406 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:40:52.866470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:52.851891 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:52.866470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:52.851956 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:53.851596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:53.851565 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 16:40:53.852084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:53.851634 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 16:40:55.188250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.188218 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:55.189947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.189922 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:55.190901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.190781 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:40:55.312727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.312688 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:40:55.317141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.317114 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.338490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.334986 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9501ab\"" Nov 03 16:40:55.338490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.337185 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:40:55.380423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.380140 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.380651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.380517 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnjrq\" (UniqueName: \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.382347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.382318 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:40:55.387991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.387968 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.403676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.403634 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9501ab-dockercfg-qdq5r\"" Nov 03 16:40:55.404142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.404119 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:40:55.514469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.499624 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.514469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.499695 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.514469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.499838 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nrb\" (UniqueName: \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.514469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.500035 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnjrq\" (UniqueName: \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.514469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.501413 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.531471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.528195 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnjrq\" (UniqueName: \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\") pod \"payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.606467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.602285 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nrb\" (UniqueName: \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.606467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.602758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.606467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.604744 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.635743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.635703 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:55.641870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.641835 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nrb\" (UniqueName: \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\") pod \"payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.706496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.706210 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:55.968633 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:55.968599 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3680ceed_3650_4ddc_8244_dcf3a1467d19.slice/crio-77eca112204e8b5988dc0bf47c2a9a1a0c7a4090717a4168263a9cab59c8c752 WatchSource:0}: Error finding container 77eca112204e8b5988dc0bf47c2a9a1a0c7a4090717a4168263a9cab59c8c752: Status 404 returned error can't find the container with id 77eca112204e8b5988dc0bf47c2a9a1a0c7a4090717a4168263a9cab59c8c752 Nov 03 16:40:55.970140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:55.970113 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:40:56.066742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.066714 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:40:56.067104 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:40:56.067080 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b6cf71_9e75_409b_bfa4_bf87227133e2.slice/crio-e149e37088889e0c0d18417ee08b898836f2d8f4abb1c1b66ab2cc4170fb96be WatchSource:0}: Error finding container e149e37088889e0c0d18417ee08b898836f2d8f4abb1c1b66ab2cc4170fb96be: Status 404 returned error can't find the container with id e149e37088889e0c0d18417ee08b898836f2d8f4abb1c1b66ab2cc4170fb96be Nov 03 16:40:56.846058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.846017 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" event={"ID":"3680ceed-3650-4ddc-8244-dcf3a1467d19","Type":"ContainerStarted","Data":"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84"} Nov 03 16:40:56.846541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.846078 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" event={"ID":"3680ceed-3650-4ddc-8244-dcf3a1467d19","Type":"ContainerStarted","Data":"77eca112204e8b5988dc0bf47c2a9a1a0c7a4090717a4168263a9cab59c8c752"} Nov 03 16:40:56.846541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.846136 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:40:56.847949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.847925 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" event={"ID":"22b6cf71-9e75-409b-bfa4-bf87227133e2","Type":"ContainerStarted","Data":"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621"} Nov 03 16:40:56.848058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.847974 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" event={"ID":"22b6cf71-9e75-409b-bfa4-bf87227133e2","Type":"ContainerStarted","Data":"e149e37088889e0c0d18417ee08b898836f2d8f4abb1c1b66ab2cc4170fb96be"} Nov 03 16:40:56.848058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.848004 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:40:56.899592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.899528 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" podStartSLOduration=1.8995113670000001 podStartE2EDuration="1.899511367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:56.896851041 +0000 UTC m=+1552.230064603" watchObservedRunningTime="2026-11-03 16:40:56.899511367 +0000 UTC m=+1552.232724936" Nov 03 16:40:56.944979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:40:56.944920 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" podStartSLOduration=1.9448874470000002 podStartE2EDuration="1.944887447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:56.940236838 +0000 UTC m=+1552.273450407" watchObservedRunningTime="2026-11-03 16:40:56.944887447 +0000 UTC m=+1552.278101020" Nov 03 16:41:03.469708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:03.469620 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:03.521633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:03.521574 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:03.547619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:03.546531 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:03.648678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:03.648616 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.039616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.039571 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.214596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.214536 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.283469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.280547 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.386524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.375490 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.476480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.470536 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.476480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.471337 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:41:04.536484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.534605 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.536484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.535313 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:41:04.568483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.552227 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:04.568483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.552637 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:41:04.679556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.666665 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:41:04.679556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:04.667409 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:41:05.261524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.245978 6001 scope.go:122] "RemoveContainer" containerID="24cfc9a15917557aa68f0f00b979cec5bb4378463ada4913f8dfd790305325f9" Nov 03 16:41:05.261524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.259576 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.306327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.306272 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.538526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.537557 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:41:05.620470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.619670 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.643477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.641397 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 16:41:05.700410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.700348 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 16:41:05.769032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.750528 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:41:05.792569 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.774750 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:41:05.890027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:05.889975 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Nov 03 16:41:06.203471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.187663 6001 scope.go:122] "RemoveContainer" containerID="bcfc02349315a22cb84cd3c77b3385ef55ef251093f3492427cb5b4bee9a82aa" Nov 03 16:41:06.230478 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:41:06.206319 6001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.378s" Nov 03 16:41:06.416659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.404146 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:41:06.416659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.307680 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:06.416659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.404333 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 16:41:06.604315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.604284 6001 scope.go:122] "RemoveContainer" containerID="86b805e4800ab9d594fcd3e04022927a6793869e65c9348d2fcccdc5a687c05c" Nov 03 16:41:06.840492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.839797 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:06.869513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.866127 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:41:06.869513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.866202 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:41:06.909768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.909737 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzr7j\"" Nov 03 16:41:06.950924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.950894 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:41:06.973471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.952005 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz7n\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.011508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:06.976118 6001 scope.go:122] "RemoveContainer" containerID="0183a530bdba3ba907177d5fd0d7f1fb9b241d54b17b4db8bf27d07e375c04c6" Nov 03 16:41:07.012911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.012868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.013251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.013223 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.013492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.013469 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.013756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.013736 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.013985 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.013966 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.014181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.014164 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.143371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.143284 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.164787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.164747 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.165041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.165018 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.165285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.165265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.165492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.165473 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.165673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.165655 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.165850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.165834 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz7n\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.167338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.167312 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.261039 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.260988 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:41:07.261764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.261721 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:41:07.262588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.262553 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:07.263199 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.260931 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:07.263647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.262472 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:07.264047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.261937 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:41:07.438515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.435827 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.438515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.436090 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:07.470483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.470290 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 16:41:07.470483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.470406 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:41:07.470483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.470466 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:41:07.527468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.526885 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:07.624763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.624658 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.630775 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.630739 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.652986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.652954 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz7n\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.671901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.671861 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:41:07.672862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.672757 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:41:07.742328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.731292 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.742328 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.731699 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-p7wz2\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.857788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.840118 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:07.874699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.851519 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:41:07.874699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.864085 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:41:07.885785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.878652 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:41:07.885785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.880090 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:41:07.885785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.884877 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:41:07.889671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.889644 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:41:07.899531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:07.899491 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 16:41:08.346622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:08.346594 6001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:41:08.349296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:08.349267 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 16:41:08.684960 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:08.684859 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" event={"ID":"8bb05426-b20c-4e5c-b4f5-0c4b44430808","Type":"ContainerStarted","Data":"da4707745f0da8efdffefb9d2cf406d41c0eaca9a96c675347988bd792d0625b"} Nov 03 16:41:11.098273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:11.098233 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:41:11.098628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:11.098308 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:41:11.715778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:11.715738 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" event={"ID":"8bb05426-b20c-4e5c-b4f5-0c4b44430808","Type":"ContainerStarted","Data":"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39"} Nov 03 16:41:11.715983 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:11.715852 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:11.797314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:11.797238 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" podStartSLOduration=5.047301437 podStartE2EDuration="7.797216885s" totalImagesPullingTime="2.749915448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:11.777877809 +0000 UTC m=+1567.111091379" watchObservedRunningTime="2026-11-03 16:41:11.797216885 +0000 UTC m=+1567.130430457" Nov 03 16:41:12.728369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:12.728339 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 16:41:17.065543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:17.065501 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799"] Nov 03 16:41:17.070724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:17.070682 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-595574b5-8k799"] Nov 03 16:41:18.790852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:18.790816 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ecd2c1-577b-47a9-afa9-e48cb0118122" path="/var/lib/kubelet/pods/b4ecd2c1-577b-47a9-afa9-e48cb0118122/volumes" Nov 03 16:41:22.252005 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:41:22.251966 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:41:24.002998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.002965 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:41:24.009366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.009338 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.035628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.035594 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:41:24.097987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.097948 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8650a6e8-openshift-default-dockercfg-dq7gn\"" Nov 03 16:41:24.118903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.118862 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.118994 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119043 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119164 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119253 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119302 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119337 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119369 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks75f\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.119540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.119524 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.220813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.220767 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.220994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.220835 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.220994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.220951 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221080 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221123 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221215 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks75f\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221271 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221326 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221366 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221436 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.221577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221521 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.222026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.221996 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.222168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.222090 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.223684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.223660 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.223795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.223780 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.246683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.246652 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.247431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.247403 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks75f\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.341424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.341334 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:24.665596 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:41:24.665556 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbe7726_a38e_491f_ab4d_6f2fdbeb9e54.slice/crio-113f12c4532f9a69157cf31b97fd3879b4bde1582fa0833c45ee6b8c83fbb5b3 WatchSource:0}: Error finding container 113f12c4532f9a69157cf31b97fd3879b4bde1582fa0833c45ee6b8c83fbb5b3: Status 404 returned error can't find the container with id 113f12c4532f9a69157cf31b97fd3879b4bde1582fa0833c45ee6b8c83fbb5b3 Nov 03 16:41:24.668063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.668039 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:41:24.668561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.668528 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:41:24.668612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.668600 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:41:24.668651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:24.668630 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:41:25.019205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.019158 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" event={"ID":"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54","Type":"ContainerStarted","Data":"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562"} Nov 03 16:41:25.021768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.019274 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" event={"ID":"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54","Type":"ContainerStarted","Data":"113f12c4532f9a69157cf31b97fd3879b4bde1582fa0833c45ee6b8c83fbb5b3"} Nov 03 16:41:25.097523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.097435 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podStartSLOduration=2.097411487 podStartE2EDuration="2.097411487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:25.078963892 +0000 UTC m=+1580.412177452" watchObservedRunningTime="2026-11-03 16:41:25.097411487 +0000 UTC m=+1580.430625063" Nov 03 16:41:25.342000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.341902 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:25.343919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.343890 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:25.344044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:25.343942 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:26.342457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:26.342414 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:26.342997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:26.342517 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:27.342037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:27.342004 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:27.342239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:27.342060 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:28.342567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:28.342531 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:28.343078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:28.342600 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:29.342520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:29.342481 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:29.342731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:29.342559 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:30.342026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:30.341991 6001 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Nov 03 16:41:30.342210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:30.342044 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Nov 03 16:41:31.347634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:31.347601 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:31.348136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:31.347995 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:31.348713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:31.348690 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:41:32.036896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:32.036858 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt"] Nov 03 16:41:32.048666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:32.048629 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44815ef-gj8vt"] Nov 03 16:41:32.792910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:32.792872 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f908ffb-6de3-4199-a9ab-1cc88b7524f3" path="/var/lib/kubelet/pods/9f908ffb-6de3-4199-a9ab-1cc88b7524f3/volumes" Nov 03 16:41:52.251105 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:41:52.251066 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:41:54.040717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:54.040679 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7"] Nov 03 16:41:54.052941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:54.052905 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6760d390-rlgl7"] Nov 03 16:41:54.791397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:41:54.791354 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0f2e06-b502-40e2-bc60-1d3741e6dd3f" path="/var/lib/kubelet/pods/df0f2e06-b502-40e2-bc60-1d3741e6dd3f/volumes" Nov 03 16:42:00.033880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:00.033840 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z"] Nov 03 16:42:00.046305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:00.046271 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db7ee809-66k2z"] Nov 03 16:42:00.790830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:00.790797 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d71caa4-66a4-4435-bdbd-7ea4fc7d1487" path="/var/lib/kubelet/pods/0d71caa4-66a4-4435-bdbd-7ea4fc7d1487/volumes" Nov 03 16:42:01.032034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:01.031997 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288"] Nov 03 16:42:01.044750 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:01.044666 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da4ca32d-ch288"] Nov 03 16:42:02.792823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:02.792782 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f" path="/var/lib/kubelet/pods/b6b2f62d-ac2e-44fa-864c-9b6e3be1fe5f/volumes" Nov 03 16:42:04.038599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:04.038554 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp"] Nov 03 16:42:04.050735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:04.050705 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2eebaf6-g5qvp"] Nov 03 16:42:04.800028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:04.799997 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1c9128-ae78-445f-94d5-20620e5fd9e8" path="/var/lib/kubelet/pods/fd1c9128-ae78-445f-94d5-20620e5fd9e8/volumes" Nov 03 16:42:08.031099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:08.031060 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt"] Nov 03 16:42:08.044198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:08.044166 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c548914b-z28lt"] Nov 03 16:42:08.791875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:08.791838 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9249befe-1afe-43e4-97b4-cb9ee45faac2" path="/var/lib/kubelet/pods/9249befe-1afe-43e4-97b4-cb9ee45faac2/volumes" Nov 03 16:42:14.039333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:14.039290 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck"] Nov 03 16:42:14.048535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:14.048501 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f3c711-gr9ck"] Nov 03 16:42:14.793654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:14.793617 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea074ca-583b-4255-8c99-970d911753e8" path="/var/lib/kubelet/pods/dea074ca-583b-4255-8c99-970d911753e8/volumes" Nov 03 16:42:17.671796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:17.671716 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:42:17.724259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:17.724220 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:42:21.037558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.037524 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t"] Nov 03 16:42:21.041100 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.041081 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.062590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.062547 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t"] Nov 03 16:42:21.069954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.069883 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:42:21.070145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.070079 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:42:21.070214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.070151 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:42:21.070396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.070371 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:42:21.187492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.187411 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.187686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.187564 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kg9x\" (UniqueName: \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.187953 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.187926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.289468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.289365 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.289468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.289435 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.289724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.289504 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kg9x\" (UniqueName: \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.290174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.290125 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.292546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.292518 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.315633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.315589 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kg9x\" (UniqueName: \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\") pod \"maas-api-revoke-sub-c3752ff2-bhn6t\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.354790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.354743 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:21.669803 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:42:21.669773 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bc1bf0_b0e6_499a_aff8_068958ac7538.slice/crio-4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d WatchSource:0}: Error finding container 4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d: Status 404 returned error can't find the container with id 4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d Nov 03 16:42:21.671401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.671373 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t"] Nov 03 16:42:21.709288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:21.709248 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" event={"ID":"e8bc1bf0-b0e6-499a-aff8-068958ac7538","Type":"ContainerStarted","Data":"4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d"} Nov 03 16:42:22.252452 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:42:22.252396 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:42:22.673049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:22.673008 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" podUID="3680ceed-3650-4ddc-8244-dcf3a1467d19" containerName="payload-processing" containerID="cri-o://057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84" gracePeriod=25 Nov 03 16:42:22.726329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:22.726273 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" podUID="22b6cf71-9e75-409b-bfa4-bf87227133e2" containerName="payload-pre-processing" containerID="cri-o://25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621" gracePeriod=25 Nov 03 16:42:22.732084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:22.732054 6001 generic.go:356] "Generic (PLEG): container finished" podID="e8bc1bf0-b0e6-499a-aff8-068958ac7538" containerID="6d0d3e91783e1daf955da41eda07f88a987a1bc4171e35e246c76cfab3510ec0" exitCode=0 Nov 03 16:42:22.732226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:22.732122 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" event={"ID":"e8bc1bf0-b0e6-499a-aff8-068958ac7538","Type":"ContainerDied","Data":"6d0d3e91783e1daf955da41eda07f88a987a1bc4171e35e246c76cfab3510ec0"} Nov 03 16:42:23.060414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.060384 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:42:23.159534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.159503 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:42:23.228761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.228727 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\") pod \"3680ceed-3650-4ddc-8244-dcf3a1467d19\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " Nov 03 16:42:23.228761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.228768 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\" (UniqueName: \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\") pod \"3680ceed-3650-4ddc-8244-dcf3a1467d19\" (UID: \"3680ceed-3650-4ddc-8244-dcf3a1467d19\") " Nov 03 16:42:23.229227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.229195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume" pod "3680ceed-3650-4ddc-8244-dcf3a1467d19" (UID: "3680ceed-3650-4ddc-8244-dcf3a1467d19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:23.229601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.229581 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3680ceed-3650-4ddc-8244-dcf3a1467d19-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:23.232212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.232184 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq" pod "3680ceed-3650-4ddc-8244-dcf3a1467d19" (UID: "3680ceed-3650-4ddc-8244-dcf3a1467d19"). InnerVolumeSpecName "kube-api-access-hnjrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:23.331179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.331137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\") pod \"22b6cf71-9e75-409b-bfa4-bf87227133e2\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " Nov 03 16:42:23.331179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.331182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\" (UniqueName: \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\") pod \"22b6cf71-9e75-409b-bfa4-bf87227133e2\" (UID: \"22b6cf71-9e75-409b-bfa4-bf87227133e2\") " Nov 03 16:42:23.331741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.331598 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume" pod "22b6cf71-9e75-409b-bfa4-bf87227133e2" (UID: "22b6cf71-9e75-409b-bfa4-bf87227133e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:23.332137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.332113 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b6cf71-9e75-409b-bfa4-bf87227133e2-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:23.332230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.332143 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnjrq\" (UniqueName: \"kubernetes.io/projected/3680ceed-3650-4ddc-8244-dcf3a1467d19-kube-api-access-hnjrq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:23.333634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.333601 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb" pod "22b6cf71-9e75-409b-bfa4-bf87227133e2" (UID: "22b6cf71-9e75-409b-bfa4-bf87227133e2"). InnerVolumeSpecName "kube-api-access-55nrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:23.434064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.433968 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55nrb\" (UniqueName: \"kubernetes.io/projected/22b6cf71-9e75-409b-bfa4-bf87227133e2-kube-api-access-55nrb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:23.742293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.742253 6001 generic.go:356] "Generic (PLEG): container finished" podID="3680ceed-3650-4ddc-8244-dcf3a1467d19" containerID="057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84" exitCode=0 Nov 03 16:42:23.742293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.742305 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" Nov 03 16:42:23.742585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.742335 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" event={"ID":"3680ceed-3650-4ddc-8244-dcf3a1467d19","Type":"ContainerDied","Data":"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84"} Nov 03 16:42:23.742585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.742392 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6" event={"ID":"3680ceed-3650-4ddc-8244-dcf3a1467d19","Type":"ContainerDied","Data":"77eca112204e8b5988dc0bf47c2a9a1a0c7a4090717a4168263a9cab59c8c752"} Nov 03 16:42:23.742585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.742414 6001 scope.go:122] "RemoveContainer" containerID="057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84" Nov 03 16:42:23.744154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.744130 6001 generic.go:356] "Generic (PLEG): container finished" podID="22b6cf71-9e75-409b-bfa4-bf87227133e2" containerID="25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621" exitCode=0 Nov 03 16:42:23.744273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.744179 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" Nov 03 16:42:23.744273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.744198 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" event={"ID":"22b6cf71-9e75-409b-bfa4-bf87227133e2","Type":"ContainerDied","Data":"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621"} Nov 03 16:42:23.744273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.744260 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg" event={"ID":"22b6cf71-9e75-409b-bfa4-bf87227133e2","Type":"ContainerDied","Data":"e149e37088889e0c0d18417ee08b898836f2d8f4abb1c1b66ab2cc4170fb96be"} Nov 03 16:42:23.756919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.756895 6001 scope.go:122] "RemoveContainer" containerID="057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84" Nov 03 16:42:23.757259 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:42:23.757235 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84\": container with ID starting with 057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84 not found: ID does not exist" containerID="057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84" Nov 03 16:42:23.757322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.757269 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84"} err="failed to get container status \"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84\": rpc error: code = NotFound desc = could not find container \"057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84\": container with ID starting with 057cf78d5096eef2dbf116c231fbf975aab40f96a72751ab92553acff4bcdf84 not found: ID does not exist" Nov 03 16:42:23.757322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.757290 6001 scope.go:122] "RemoveContainer" containerID="25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621" Nov 03 16:42:23.770274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.770004 6001 scope.go:122] "RemoveContainer" containerID="25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621" Nov 03 16:42:23.770671 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:42:23.770631 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621\": container with ID starting with 25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621 not found: ID does not exist" containerID="25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621" Nov 03 16:42:23.770788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.770677 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621"} err="failed to get container status \"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621\": rpc error: code = NotFound desc = could not find container \"25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621\": container with ID starting with 25c5f0c01e9c951e0134458661d36acede38e483dbfa76e1585b6bc3eac67621 not found: ID does not exist" Nov 03 16:42:23.801117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.801081 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:42:23.815094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.815058 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9501ab-9bbc77b79-982gg"] Nov 03 16:42:23.854544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.854510 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:42:23.859934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.859892 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9501ab-7fb969d59f-nqmb6"] Nov 03 16:42:23.928913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:23.928886 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:24.042772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.042677 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\") pod \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " Nov 03 16:42:24.042937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.042821 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\") pod \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " Nov 03 16:42:24.042999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.042965 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\" (UniqueName: \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\") pod \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\" (UID: \"e8bc1bf0-b0e6-499a-aff8-068958ac7538\") " Nov 03 16:42:24.043150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.043121 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca" pod "e8bc1bf0-b0e6-499a-aff8-068958ac7538" (UID: "e8bc1bf0-b0e6-499a-aff8-068958ac7538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:24.043599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.043568 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:24.045009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.044989 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca" pod "e8bc1bf0-b0e6-499a-aff8-068958ac7538" (UID: "e8bc1bf0-b0e6-499a-aff8-068958ac7538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:24.045101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.045056 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x" pod "e8bc1bf0-b0e6-499a-aff8-068958ac7538" (UID: "e8bc1bf0-b0e6-499a-aff8-068958ac7538"). InnerVolumeSpecName "kube-api-access-8kg9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:24.145345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.145305 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8bc1bf0-b0e6-499a-aff8-068958ac7538-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:24.145345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.145337 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kg9x\" (UniqueName: \"kubernetes.io/projected/e8bc1bf0-b0e6-499a-aff8-068958ac7538-kube-api-access-8kg9x\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:24.756920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.756889 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" Nov 03 16:42:24.757761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.757610 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t" event={"ID":"e8bc1bf0-b0e6-499a-aff8-068958ac7538","Type":"ContainerDied","Data":"4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d"} Nov 03 16:42:24.757761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.757661 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b74b6f4044f75a584bd9a92232e4ea819b8dee0cb4e0f41cdbae2bd6e588c4d" Nov 03 16:42:24.801220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.801187 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b6cf71-9e75-409b-bfa4-bf87227133e2" path="/var/lib/kubelet/pods/22b6cf71-9e75-409b-bfa4-bf87227133e2/volumes" Nov 03 16:42:24.801754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:24.801738 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3680ceed-3650-4ddc-8244-dcf3a1467d19" path="/var/lib/kubelet/pods/3680ceed-3650-4ddc-8244-dcf3a1467d19/volumes" Nov 03 16:42:42.033746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:42.033703 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j"] Nov 03 16:42:42.046316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:42.046284 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a056ea95-hxc5j"] Nov 03 16:42:42.792333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:42.792292 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0880448a-0f2a-411e-a56d-be8e21ac6539" path="/var/lib/kubelet/pods/0880448a-0f2a-411e-a56d-be8e21ac6539/volumes" Nov 03 16:42:52.252731 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:42:52.252692 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:42:55.605765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.605730 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks"] Nov 03 16:42:55.606498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606420 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3680ceed-3650-4ddc-8244-dcf3a1467d19" containerName="payload-processing" Nov 03 16:42:55.606498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606436 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3680ceed-3650-4ddc-8244-dcf3a1467d19" containerName="payload-processing" Nov 03 16:42:55.606600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606502 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e8bc1bf0-b0e6-499a-aff8-068958ac7538" containerName="revoke-keys" Nov 03 16:42:55.606600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606509 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bc1bf0-b0e6-499a-aff8-068958ac7538" containerName="revoke-keys" Nov 03 16:42:55.606600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606528 6001 cpu_manager.go:414] "containerMap: removing container" podUID="22b6cf71-9e75-409b-bfa4-bf87227133e2" containerName="payload-pre-processing" Nov 03 16:42:55.606600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606533 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b6cf71-9e75-409b-bfa4-bf87227133e2" containerName="payload-pre-processing" Nov 03 16:42:55.606728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606646 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3680ceed-3650-4ddc-8244-dcf3a1467d19" containerName="payload-processing" Nov 03 16:42:55.606728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606660 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bc1bf0-b0e6-499a-aff8-068958ac7538" containerName="revoke-keys" Nov 03 16:42:55.606728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.606669 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b6cf71-9e75-409b-bfa4-bf87227133e2" containerName="payload-pre-processing" Nov 03 16:42:55.609323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.609306 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.626968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.626940 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:42:55.640004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.639970 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:42:55.640367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.640349 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:42:55.641525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.641507 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:42:55.650561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.650526 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks"] Nov 03 16:42:55.733637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.733599 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.733842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.733821 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.733948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.733926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv4z8\" (UniqueName: \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.835397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.835345 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.835603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.835456 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv4z8\" (UniqueName: \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.835603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.835582 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.836139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.836118 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.838056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.838038 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.866670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.866600 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv4z8\" (UniqueName: \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\") pod \"maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:55.925139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:55.925100 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:56.346756 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:42:56.346719 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967608d3_9f07_4c07_aef1_7047e0e4b5bc.slice/crio-7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4 WatchSource:0}: Error finding container 7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4: Status 404 returned error can't find the container with id 7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4 Nov 03 16:42:56.348621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:56.348593 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks"] Nov 03 16:42:57.112390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:57.112348 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" event={"ID":"967608d3-9f07-4c07-aef1-7047e0e4b5bc","Type":"ContainerStarted","Data":"a125a8ddcc9d0aad627d3973f2ca435802ae7879945878cc25c8819f97a74ce2"} Nov 03 16:42:57.115739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:57.112412 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" event={"ID":"967608d3-9f07-4c07-aef1-7047e0e4b5bc","Type":"ContainerStarted","Data":"7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4"} Nov 03 16:42:57.166666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:57.166604 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" podStartSLOduration=2.166587061 podStartE2EDuration="2.166587061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:57.161013275 +0000 UTC m=+1672.494226844" watchObservedRunningTime="2026-11-03 16:42:57.166587061 +0000 UTC m=+1672.499800631" Nov 03 16:42:58.123158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:58.123119 6001 generic.go:356] "Generic (PLEG): container finished" podID="967608d3-9f07-4c07-aef1-7047e0e4b5bc" containerID="a125a8ddcc9d0aad627d3973f2ca435802ae7879945878cc25c8819f97a74ce2" exitCode=0 Nov 03 16:42:58.123637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:58.123206 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" event={"ID":"967608d3-9f07-4c07-aef1-7047e0e4b5bc","Type":"ContainerDied","Data":"a125a8ddcc9d0aad627d3973f2ca435802ae7879945878cc25c8819f97a74ce2"} Nov 03 16:42:59.288766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.288735 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:42:59.413969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.413885 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\") pod \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " Nov 03 16:42:59.413969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.413920 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\") pod \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " Nov 03 16:42:59.414166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.414050 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\" (UniqueName: \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\") pod \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\" (UID: \"967608d3-9f07-4c07-aef1-7047e0e4b5bc\") " Nov 03 16:42:59.414313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.414287 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca" pod "967608d3-9f07-4c07-aef1-7047e0e4b5bc" (UID: "967608d3-9f07-4c07-aef1-7047e0e4b5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:59.414833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.414814 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:59.416036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.416018 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca" pod "967608d3-9f07-4c07-aef1-7047e0e4b5bc" (UID: "967608d3-9f07-4c07-aef1-7047e0e4b5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:42:59.416311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.416168 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8" pod "967608d3-9f07-4c07-aef1-7047e0e4b5bc" (UID: "967608d3-9f07-4c07-aef1-7047e0e4b5bc"). InnerVolumeSpecName "kube-api-access-jv4z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:59.516875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.516836 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967608d3-9f07-4c07-aef1-7047e0e4b5bc-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:42:59.516875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:42:59.516865 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv4z8\" (UniqueName: \"kubernetes.io/projected/967608d3-9f07-4c07-aef1-7047e0e4b5bc-kube-api-access-jv4z8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:00.144018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:00.143985 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" Nov 03 16:43:00.144232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:00.144017 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks" event={"ID":"967608d3-9f07-4c07-aef1-7047e0e4b5bc","Type":"ContainerDied","Data":"7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4"} Nov 03 16:43:00.144232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:00.144078 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b1ade6c6ab99903797c0603fb203059d901002bf4116a1601ee338e86aef9d4" Nov 03 16:43:04.918600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:04.918547 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:43:04.919265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:04.919222 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.284472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.271358 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.532728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.532612 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.767470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.763958 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.767470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.764404 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:43:05.794474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.790678 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:43:05.802634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.802582 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:05.940467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:05.939557 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:06.689466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.684645 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn"] Nov 03 16:43:06.689466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.685865 6001 cpu_manager.go:414] "containerMap: removing container" podUID="967608d3-9f07-4c07-aef1-7047e0e4b5bc" containerName="revoke-keys" Nov 03 16:43:06.689466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.685888 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967608d3-9f07-4c07-aef1-7047e0e4b5bc" containerName="revoke-keys" Nov 03 16:43:06.689466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.686069 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="967608d3-9f07-4c07-aef1-7047e0e4b5bc" containerName="revoke-keys" Nov 03 16:43:06.700374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.700337 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.717781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.717742 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkvf\" (UniqueName: \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.718217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.718194 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.718972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.718948 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.724506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.724403 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn"] Nov 03 16:43:06.732037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.731809 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:43:06.732037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.731885 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:43:06.799150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.799120 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:43:06.799388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.799179 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:43:06.820149 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.820113 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.821216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.821191 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkvf\" (UniqueName: \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.821634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.821614 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.823609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.823586 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.842642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.841914 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:06.855943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:06.855902 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkvf\" (UniqueName: \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\") pod \"maas-api-revoke-sub-1ee5c5d4-dhpdn\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:07.050747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:07.050707 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:07.395471 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:43:07.395422 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a697ecc_347b_4296_a3d1_c36d9d4c68ee.slice/crio-dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597 WatchSource:0}: Error finding container dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597: Status 404 returned error can't find the container with id dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597 Nov 03 16:43:07.396435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:07.396404 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn"] Nov 03 16:43:08.013252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.013168 6001 scope.go:122] "RemoveContainer" containerID="443b63832516dcf0685bfc0957b43b6eb757186af618d6f2c00aec9844b9d081" Nov 03 16:43:08.025323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.025302 6001 scope.go:122] "RemoveContainer" containerID="76dc337b66505f2e6daa2eb46bb0ff678cad55ca684271e3c4e9871de7b8d0ce" Nov 03 16:43:08.035790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.035761 6001 scope.go:122] "RemoveContainer" containerID="2a26d47f3544f891417be93bdc4d49a0ba8752a3e8d1bb9872bcd81e5430f803" Nov 03 16:43:08.046433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.046405 6001 scope.go:122] "RemoveContainer" containerID="a480a484bd241430cfdd130ea481066b86274faaf5c4d8cfdc4e548a357a25f8" Nov 03 16:43:08.057128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.056984 6001 scope.go:122] "RemoveContainer" containerID="f4597fff32c0c307f78c1a2e900b6ef8752167e0bd545ceb0d5e83eb684e77d9" Nov 03 16:43:08.067852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.067828 6001 scope.go:122] "RemoveContainer" containerID="80cf8c1e557d06f44bf19270a4247d796c645f121108654815ea891c02922669" Nov 03 16:43:08.078276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.078254 6001 scope.go:122] "RemoveContainer" containerID="9f144061c5ee6043ce443feb34d6dbaf13e403b596d1856c7803f32c593f9e5c" Nov 03 16:43:08.343292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.343203 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" event={"ID":"2a697ecc-347b-4296-a3d1-c36d9d4c68ee","Type":"ContainerStarted","Data":"8808403a19a5970faa22b64254316ffdc12c29642365c3ca853d94997655e0e7"} Nov 03 16:43:08.343292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.343265 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" event={"ID":"2a697ecc-347b-4296-a3d1-c36d9d4c68ee","Type":"ContainerStarted","Data":"dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597"} Nov 03 16:43:08.409373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:08.409319 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" podStartSLOduration=2.409302428 podStartE2EDuration="2.409302428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:08.407392209 +0000 UTC m=+1683.740605789" watchObservedRunningTime="2026-11-03 16:43:08.409302428 +0000 UTC m=+1683.742515997" Nov 03 16:43:09.353426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:09.353384 6001 generic.go:356] "Generic (PLEG): container finished" podID="2a697ecc-347b-4296-a3d1-c36d9d4c68ee" containerID="8808403a19a5970faa22b64254316ffdc12c29642365c3ca853d94997655e0e7" exitCode=0 Nov 03 16:43:09.353872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:09.353476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" event={"ID":"2a697ecc-347b-4296-a3d1-c36d9d4c68ee","Type":"ContainerDied","Data":"8808403a19a5970faa22b64254316ffdc12c29642365c3ca853d94997655e0e7"} Nov 03 16:43:10.537078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.537050 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:10.622990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.622884 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\") pod \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " Nov 03 16:43:10.623182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.623001 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\" (UniqueName: \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\") pod \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " Nov 03 16:43:10.623182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.623171 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\") pod \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\" (UID: \"2a697ecc-347b-4296-a3d1-c36d9d4c68ee\") " Nov 03 16:43:10.623541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.623504 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca" pod "2a697ecc-347b-4296-a3d1-c36d9d4c68ee" (UID: "2a697ecc-347b-4296-a3d1-c36d9d4c68ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:10.624041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.624006 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:10.625689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.625657 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf" pod "2a697ecc-347b-4296-a3d1-c36d9d4c68ee" (UID: "2a697ecc-347b-4296-a3d1-c36d9d4c68ee"). InnerVolumeSpecName "kube-api-access-rkkvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:10.626052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.626022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca" pod "2a697ecc-347b-4296-a3d1-c36d9d4c68ee" (UID: "2a697ecc-347b-4296-a3d1-c36d9d4c68ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:10.725704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.725664 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:10.725704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:10.725705 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkvf\" (UniqueName: \"kubernetes.io/projected/2a697ecc-347b-4296-a3d1-c36d9d4c68ee-kube-api-access-rkkvf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:11.370914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:11.370881 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" Nov 03 16:43:11.371109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:11.370913 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn" event={"ID":"2a697ecc-347b-4296-a3d1-c36d9d4c68ee","Type":"ContainerDied","Data":"dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597"} Nov 03 16:43:11.371109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:11.370964 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dffe250e2ec0917fabb69f1be3d4cd662ad03a44703997f96f6c08cb7481a597" Nov 03 16:43:14.515219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.515182 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd"] Nov 03 16:43:14.515925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.515909 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2a697ecc-347b-4296-a3d1-c36d9d4c68ee" containerName="revoke-keys" Nov 03 16:43:14.515976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.515927 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a697ecc-347b-4296-a3d1-c36d9d4c68ee" containerName="revoke-keys" Nov 03 16:43:14.516079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.516068 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a697ecc-347b-4296-a3d1-c36d9d4c68ee" containerName="revoke-keys" Nov 03 16:43:14.519876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.519853 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.539545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.539515 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd"] Nov 03 16:43:14.547731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.547701 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:43:14.547731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.547722 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:43:14.547948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.547750 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:43:14.550651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.550631 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:43:14.594181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.594138 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.594378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.594306 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.594529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.594508 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqcs\" (UniqueName: \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.695730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.695689 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.695928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.695811 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.695981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.695945 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqcs\" (UniqueName: \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.696542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.696519 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.698197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.698176 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.722525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.722489 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqcs\" (UniqueName: \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\") pod \"maas-api-revoke-sub-5608dbd3-g6rdd\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:14.834925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:14.834838 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:15.173995 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:43:15.173962 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0311d3_08a8_46e9_a80e_182ab828be9c.slice/crio-fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87 WatchSource:0}: Error finding container fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87: Status 404 returned error can't find the container with id fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87 Nov 03 16:43:15.177281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:15.177250 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd"] Nov 03 16:43:15.404152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:15.404113 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" event={"ID":"3b0311d3-08a8-46e9-a80e-182ab828be9c","Type":"ContainerStarted","Data":"fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87"} Nov 03 16:43:16.414079 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:16.414033 6001 generic.go:356] "Generic (PLEG): container finished" podID="3b0311d3-08a8-46e9-a80e-182ab828be9c" containerID="cc7e5c5aae2afd6f5e443baa133901fa556049f0eb3dc2e2d8f3184a835ab7a3" exitCode=0 Nov 03 16:43:16.414551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:16.414126 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" event={"ID":"3b0311d3-08a8-46e9-a80e-182ab828be9c","Type":"ContainerDied","Data":"cc7e5c5aae2afd6f5e443baa133901fa556049f0eb3dc2e2d8f3184a835ab7a3"} Nov 03 16:43:17.597254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.597219 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:17.763646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.763609 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\" (UniqueName: \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\") pod \"3b0311d3-08a8-46e9-a80e-182ab828be9c\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " Nov 03 16:43:17.764071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.764051 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\") pod \"3b0311d3-08a8-46e9-a80e-182ab828be9c\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " Nov 03 16:43:17.764208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.764193 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\") pod \"3b0311d3-08a8-46e9-a80e-182ab828be9c\" (UID: \"3b0311d3-08a8-46e9-a80e-182ab828be9c\") " Nov 03 16:43:17.766148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.766110 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca" pod "3b0311d3-08a8-46e9-a80e-182ab828be9c" (UID: "3b0311d3-08a8-46e9-a80e-182ab828be9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:17.787530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.778212 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs" pod "3b0311d3-08a8-46e9-a80e-182ab828be9c" (UID: "3b0311d3-08a8-46e9-a80e-182ab828be9c"). InnerVolumeSpecName "kube-api-access-ppqcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:17.788164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.788104 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca" pod "3b0311d3-08a8-46e9-a80e-182ab828be9c" (UID: "3b0311d3-08a8-46e9-a80e-182ab828be9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:17.867811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.867775 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppqcs\" (UniqueName: \"kubernetes.io/projected/3b0311d3-08a8-46e9-a80e-182ab828be9c-kube-api-access-ppqcs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:17.867811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.867811 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:17.868030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:17.867827 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b0311d3-08a8-46e9-a80e-182ab828be9c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:18.438823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:18.438781 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" event={"ID":"3b0311d3-08a8-46e9-a80e-182ab828be9c","Type":"ContainerDied","Data":"fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87"} Nov 03 16:43:18.438823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:18.438807 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd" Nov 03 16:43:18.439052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:18.438869 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc3fefc4284e07f57823ca7d6f86f6a65faf50b62e13c201f9e61da34ff58e87" Nov 03 16:43:22.255550 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:43:22.255517 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:43:23.031318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:23.031278 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5"] Nov 03 16:43:23.043837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:23.043801 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33fe5226-vp6r5"] Nov 03 16:43:24.794135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:24.794099 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54a87945-c57f-4250-bd3b-f13493141b24" path="/var/lib/kubelet/pods/54a87945-c57f-4250-bd3b-f13493141b24/volumes" Nov 03 16:43:27.460883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.460839 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x"] Nov 03 16:43:27.462351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.462323 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3b0311d3-08a8-46e9-a80e-182ab828be9c" containerName="revoke-keys" Nov 03 16:43:27.462560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.462542 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b0311d3-08a8-46e9-a80e-182ab828be9c" containerName="revoke-keys" Nov 03 16:43:27.463137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.463115 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b0311d3-08a8-46e9-a80e-182ab828be9c" containerName="revoke-keys" Nov 03 16:43:27.467204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.467185 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.482697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.482673 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x"] Nov 03 16:43:27.483367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.483351 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:43:27.489242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.489212 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:43:27.489350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.489237 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:43:27.496087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.496063 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:43:27.532219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.532186 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rst5\" (UniqueName: \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.532392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.532264 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.532482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.532461 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.633605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.633565 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.633788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.633684 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rst5\" (UniqueName: \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.633788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.633742 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.634367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.634341 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.636207 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.659796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.659765 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rst5\" (UniqueName: \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\") pod \"maas-api-revoke-sub-36f64772-z2n5x\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:27.784930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:27.784890 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:28.207627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:28.206365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x"] Nov 03 16:43:28.285272 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:43:28.276831 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c50c16_3bea_4747_9f91_e77f71e18fa2.slice/crio-8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436 WatchSource:0}: Error finding container 8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436: Status 404 returned error can't find the container with id 8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436 Nov 03 16:43:28.570807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:28.570773 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" event={"ID":"e9c50c16-3bea-4747-9f91-e77f71e18fa2","Type":"ContainerStarted","Data":"8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436"} Nov 03 16:43:29.629490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:29.627962 6001 generic.go:356] "Generic (PLEG): container finished" podID="e9c50c16-3bea-4747-9f91-e77f71e18fa2" containerID="f4c9b6e655c7fce33d2419da42958b8d0fa05d99df119a132142356e00f1d7da" exitCode=0 Nov 03 16:43:29.629490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:29.628173 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" event={"ID":"e9c50c16-3bea-4747-9f91-e77f71e18fa2","Type":"ContainerDied","Data":"f4c9b6e655c7fce33d2419da42958b8d0fa05d99df119a132142356e00f1d7da"} Nov 03 16:43:30.369601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:30.369553 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Nov 03 16:43:30.778469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:30.767862 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:43:30.803471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:30.783532 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:30.803471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:30.785818 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:30.883467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:30.880544 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:31.397573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.397541 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:31.443467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.443107 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\" (UniqueName: \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\") pod \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " Nov 03 16:43:31.443467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.443337 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\") pod \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " Nov 03 16:43:31.443467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.443386 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\") pod \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\" (UID: \"e9c50c16-3bea-4747-9f91-e77f71e18fa2\") " Nov 03 16:43:31.447519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.445459 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca" pod "e9c50c16-3bea-4747-9f91-e77f71e18fa2" (UID: "e9c50c16-3bea-4747-9f91-e77f71e18fa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:31.477603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.477561 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca" pod "e9c50c16-3bea-4747-9f91-e77f71e18fa2" (UID: "e9c50c16-3bea-4747-9f91-e77f71e18fa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:43:31.478636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.478598 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5" pod "e9c50c16-3bea-4747-9f91-e77f71e18fa2" (UID: "e9c50c16-3bea-4747-9f91-e77f71e18fa2"). InnerVolumeSpecName "kube-api-access-2rst5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:31.545900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.545853 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rst5\" (UniqueName: \"kubernetes.io/projected/e9c50c16-3bea-4747-9f91-e77f71e18fa2-kube-api-access-2rst5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:31.545900 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.545900 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:31.546112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.545919 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9c50c16-3bea-4747-9f91-e77f71e18fa2-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:43:31.678591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.678477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" event={"ID":"e9c50c16-3bea-4747-9f91-e77f71e18fa2","Type":"ContainerDied","Data":"8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436"} Nov 03 16:43:31.678591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.678510 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x" Nov 03 16:43:31.678817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:31.678596 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b717ee137bf0a315b3d143e49abba12fdefa62e7a302eff963e3be09f9a8436" Nov 03 16:43:36.545728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:36.544336 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks"] Nov 03 16:43:36.554642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:36.554609 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8650a6e8-5jfks"] Nov 03 16:43:36.800098 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:36.800016 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967608d3-9f07-4c07-aef1-7047e0e4b5bc" path="/var/lib/kubelet/pods/967608d3-9f07-4c07-aef1-7047e0e4b5bc/volumes" Nov 03 16:43:51.049170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:51.049073 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp"] Nov 03 16:43:51.062647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:51.062613 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd8bad3-z9kmp"] Nov 03 16:43:52.252996 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:43:52.252947 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:43:52.795267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:43:52.795226 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8495e090-6715-4250-8022-11f0e88858c4" path="/var/lib/kubelet/pods/8495e090-6715-4250-8022-11f0e88858c4/volumes" Nov 03 16:44:07.066849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:07.066811 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv"] Nov 03 16:44:07.078715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:07.078683 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526f23b9-9c9gv"] Nov 03 16:44:08.212716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.212678 6001 scope.go:122] "RemoveContainer" containerID="d613c4f35fdfad495628363ff9a9bf364aeff65f49826d831b8afbaece611aa1" Nov 03 16:44:08.223792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.223769 6001 scope.go:122] "RemoveContainer" containerID="64d298fb7bd0c595a907aea76042e07ef5341635753d9e2c17b8c3fed96c6beb" Nov 03 16:44:08.792534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.792497 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="232c95d0-49e5-4687-a79f-e750626b221b" path="/var/lib/kubelet/pods/232c95d0-49e5-4687-a79f-e750626b221b/volumes" Nov 03 16:44:08.979812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.979770 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:44:08.980500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.980480 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e9c50c16-3bea-4747-9f91-e77f71e18fa2" containerName="revoke-keys" Nov 03 16:44:08.980500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.980496 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9c50c16-3bea-4747-9f91-e77f71e18fa2" containerName="revoke-keys" Nov 03 16:44:08.980689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.980599 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9c50c16-3bea-4747-9f91-e77f71e18fa2" containerName="revoke-keys" Nov 03 16:44:08.985883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:08.985859 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.013054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.012993 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-db3c6b\"" Nov 03 16:44:09.013778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.013750 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-db3c6b-dockercfg-srngx\"" Nov 03 16:44:09.015075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.015046 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:44:09.048293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.048195 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:44:09.054122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.054091 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.078646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.078615 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:44:09.085903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.085854 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.086136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.086039 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpj44\" (UniqueName: \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.208465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.207434 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.208465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.207530 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.208465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.207605 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwmr\" (UniqueName: \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.208465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.207721 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpj44\" (UniqueName: \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.218553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.209097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.322518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.321584 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.322518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.322222 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpj44\" (UniqueName: \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\") pod \"payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.322518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.322411 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.322843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.322569 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwmr\" (UniqueName: \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.355351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.352504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwmr\" (UniqueName: \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\") pod \"payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.375828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.375791 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:09.603773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.603681 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:09.700395 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:44:09.700363 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634e4f1d_2213_4eae_b7c1_8ac92671cd59.slice/crio-d157a122fbb3d670a7cc79de37f06bad52a60ae1db4037601bcd7920047b97be WatchSource:0}: Error finding container d157a122fbb3d670a7cc79de37f06bad52a60ae1db4037601bcd7920047b97be: Status 404 returned error can't find the container with id d157a122fbb3d670a7cc79de37f06bad52a60ae1db4037601bcd7920047b97be Nov 03 16:44:09.701218 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.701193 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:44:09.913593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:09.913559 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:44:09.916541 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:44:09.916502 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ddb60f_6ac9_4282_ad23_5b99bbcea118.slice/crio-0b036f0ac60cd1c1c0f805c00e797caa79095e9e71f2714056679a905f868bff WatchSource:0}: Error finding container 0b036f0ac60cd1c1c0f805c00e797caa79095e9e71f2714056679a905f868bff: Status 404 returned error can't find the container with id 0b036f0ac60cd1c1c0f805c00e797caa79095e9e71f2714056679a905f868bff Nov 03 16:44:10.339419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.339359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" event={"ID":"38ddb60f-6ac9-4282-ad23-5b99bbcea118","Type":"ContainerStarted","Data":"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed"} Nov 03 16:44:10.340056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.339431 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" event={"ID":"38ddb60f-6ac9-4282-ad23-5b99bbcea118","Type":"ContainerStarted","Data":"0b036f0ac60cd1c1c0f805c00e797caa79095e9e71f2714056679a905f868bff"} Nov 03 16:44:10.340699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.340673 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:10.353509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.353478 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" event={"ID":"634e4f1d-2213-4eae-b7c1-8ac92671cd59","Type":"ContainerStarted","Data":"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce"} Nov 03 16:44:10.353736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.353722 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" event={"ID":"634e4f1d-2213-4eae-b7c1-8ac92671cd59","Type":"ContainerStarted","Data":"d157a122fbb3d670a7cc79de37f06bad52a60ae1db4037601bcd7920047b97be"} Nov 03 16:44:10.353829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.353818 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:10.419645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.419592 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" podStartSLOduration=2.419578149 podStartE2EDuration="2.419578149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:10.417337588 +0000 UTC m=+1745.750551158" watchObservedRunningTime="2026-11-03 16:44:10.419578149 +0000 UTC m=+1745.752791719" Nov 03 16:44:10.471652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:10.471531 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" podStartSLOduration=2.471509715 podStartE2EDuration="2.471509715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:10.464985285 +0000 UTC m=+1745.798198856" watchObservedRunningTime="2026-11-03 16:44:10.471509715 +0000 UTC m=+1745.804723285" Nov 03 16:44:12.660674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.659401 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:44:12.681469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.677799 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:12.743560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.743478 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:44:12.789708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.784152 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-73cc79-dockercfg-wk9dg\"" Nov 03 16:44:12.789708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.784317 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-73cc79\"" Nov 03 16:44:12.933512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.924350 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:12.933512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:12.924736 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxcvl\" (UniqueName: \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.059469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.047380 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.059469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.047662 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcvl\" (UniqueName: \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.059469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.058349 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.098803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.095125 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:44:13.186529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.181595 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcvl\" (UniqueName: \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\") pod \"payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.210469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.210126 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:44:13.210469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.210254 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.298469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.297782 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:13.386177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.385465 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhpc2\" (UniqueName: \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.386177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.385668 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.565484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.492578 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.565484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.492858 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhpc2\" (UniqueName: \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.565484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.496071 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.711810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.711690 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhpc2\" (UniqueName: \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\") pod \"payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.890352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.889155 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:13.925822 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:44:13.925786 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7665b3fc_6789_4027_bdc7_a4c82f9fb085.slice/crio-512555d1ff6c2ff52577b3865c994b68bac8383dc3da91be580e4faa298fbaef WatchSource:0}: Error finding container 512555d1ff6c2ff52577b3865c994b68bac8383dc3da91be580e4faa298fbaef: Status 404 returned error can't find the container with id 512555d1ff6c2ff52577b3865c994b68bac8383dc3da91be580e4faa298fbaef Nov 03 16:44:13.928523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:13.927865 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:44:14.256840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.256806 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:44:14.256980 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:44:14.256943 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0c349c_c6ae_4a92_8c92_80ee333ad763.slice/crio-df131493cd92e942c4e9cdd05e447f0a2e9d09861faaac1abbc5a3216e4414d7 WatchSource:0}: Error finding container df131493cd92e942c4e9cdd05e447f0a2e9d09861faaac1abbc5a3216e4414d7: Status 404 returned error can't find the container with id df131493cd92e942c4e9cdd05e447f0a2e9d09861faaac1abbc5a3216e4414d7 Nov 03 16:44:14.741521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.741481 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" event={"ID":"5e0c349c-c6ae-4a92-8c92-80ee333ad763","Type":"ContainerStarted","Data":"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194"} Nov 03 16:44:14.741982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.741545 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" event={"ID":"5e0c349c-c6ae-4a92-8c92-80ee333ad763","Type":"ContainerStarted","Data":"df131493cd92e942c4e9cdd05e447f0a2e9d09861faaac1abbc5a3216e4414d7"} Nov 03 16:44:14.741982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.741610 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:14.743384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.743354 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" event={"ID":"7665b3fc-6789-4027-bdc7-a4c82f9fb085","Type":"ContainerStarted","Data":"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd"} Nov 03 16:44:14.743519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.743408 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" event={"ID":"7665b3fc-6789-4027-bdc7-a4c82f9fb085","Type":"ContainerStarted","Data":"512555d1ff6c2ff52577b3865c994b68bac8383dc3da91be580e4faa298fbaef"} Nov 03 16:44:14.743519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.743459 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:14.787045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.786992 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" podStartSLOduration=2.786977001 podStartE2EDuration="2.786977001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:14.784588597 +0000 UTC m=+1750.117802166" watchObservedRunningTime="2026-11-03 16:44:14.786977001 +0000 UTC m=+1750.120190570" Nov 03 16:44:14.872396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:14.872341 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" podStartSLOduration=2.872324006 podStartE2EDuration="2.872324006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:14.87045771 +0000 UTC m=+1750.203671277" watchObservedRunningTime="2026-11-03 16:44:14.872324006 +0000 UTC m=+1750.205537575" Nov 03 16:44:21.033105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:21.033068 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt"] Nov 03 16:44:21.045602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:21.045564 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn"] Nov 03 16:44:21.050093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:21.050053 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-043be1a2-s97vt"] Nov 03 16:44:21.054329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:21.054300 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac4787b7-lkjpn"] Nov 03 16:44:21.366084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:21.366006 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:44:22.251553 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:44:22.251517 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:44:22.664154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:22.664068 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:44:22.793101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:22.793067 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49360ea3-82dc-42ed-8793-3be8ff47b9c9" path="/var/lib/kubelet/pods/49360ea3-82dc-42ed-8793-3be8ff47b9c9/volumes" Nov 03 16:44:22.793672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:22.793656 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c74aaa-e759-4ceb-950d-8d3f2a8b3895" path="/var/lib/kubelet/pods/d0c74aaa-e759-4ceb-950d-8d3f2a8b3895/volumes" Nov 03 16:44:25.759794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:25.759762 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:44:25.762036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:25.762014 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:44:30.043434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.043398 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz"] Nov 03 16:44:30.055937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.055902 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm"] Nov 03 16:44:30.060459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.060420 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a6b7e5-86dnz"] Nov 03 16:44:30.064701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.064673 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf151c15-k9qpm"] Nov 03 16:44:30.792484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.792431 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0984622a-9487-4288-8639-262d4e08f2e5" path="/var/lib/kubelet/pods/0984622a-9487-4288-8639-262d4e08f2e5/volumes" Nov 03 16:44:30.793028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:30.793015 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be00747-f424-4825-b8d6-4f9ba7e1a571" path="/var/lib/kubelet/pods/8be00747-f424-4825-b8d6-4f9ba7e1a571/volumes" Nov 03 16:44:43.033965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:43.033914 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp"] Nov 03 16:44:43.046815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:43.046783 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7907f6a-j5vcp"] Nov 03 16:44:44.799701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:44.799664 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee62d86-cdad-4f50-8279-d9c2646b765c" path="/var/lib/kubelet/pods/8ee62d86-cdad-4f50-8279-d9c2646b765c/volumes" Nov 03 16:44:47.065759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:47.065727 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr"] Nov 03 16:44:47.078581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:47.078540 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b62b149-r6bcr"] Nov 03 16:44:48.792037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:48.792005 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f3f7cb-0764-4c77-8380-afba76c25f7c" path="/var/lib/kubelet/pods/d9f3f7cb-0764-4c77-8380-afba76c25f7c/volumes" Nov 03 16:44:49.112936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:49.112855 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr"] Nov 03 16:44:49.125056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:49.125032 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95bf16e4-zwcbr"] Nov 03 16:44:50.882469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:50.867741 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50d53fd1-efa7-4ca7-b113-d84d18e14038" path="/var/lib/kubelet/pods/50d53fd1-efa7-4ca7-b113-d84d18e14038/volumes" Nov 03 16:44:52.252719 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:44:52.252676 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:44:55.005211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.005177 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:44:55.011135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.011114 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.038507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.038464 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:44:55.062512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.049827 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-156c2a\"" Nov 03 16:44:55.062512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.050539 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-156c2a-dockercfg-bnpps\"" Nov 03 16:44:55.138496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.130639 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.138496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.130764 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jbp\" (UniqueName: \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.232063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.232021 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.232248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.232146 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jbp\" (UniqueName: \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.232965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.232939 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.259251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.259179 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jbp\" (UniqueName: \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\") pod \"payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.330475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.330414 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:55.643779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:55.643747 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:44:56.162755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:56.162714 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" event={"ID":"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f","Type":"ContainerStarted","Data":"42247c809e8b5b5e1efc4fccda4600bfe56d49844a658b938dbd48b5d974ad4c"} Nov 03 16:44:58.199261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:58.199215 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" event={"ID":"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f","Type":"ContainerStarted","Data":"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652"} Nov 03 16:44:58.199710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:58.199328 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:44:58.256388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:58.256326 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" podStartSLOduration=2.630971113 podStartE2EDuration="4.256310651s" totalImagesPullingTime="1.625339538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:58.253945418 +0000 UTC m=+1793.587159013" watchObservedRunningTime="2026-11-03 16:44:58.256310651 +0000 UTC m=+1793.589524220" Nov 03 16:44:59.067206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:59.067167 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9"] Nov 03 16:44:59.081359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:44:59.081318 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-406bd897-669f9"] Nov 03 16:45:00.052481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.052405 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f"] Nov 03 16:45:00.063342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.063308 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e469e595-b9d4f"] Nov 03 16:45:00.137382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.137341 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:45:00.142043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.142016 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.168122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.168099 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2aa2ad\"" Nov 03 16:45:00.179330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.179294 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:45:00.207830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.207791 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:45:00.212884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.212853 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.253464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.253415 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:45:00.266620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.266579 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2aa2ad-dockercfg-2t42j\"" Nov 03 16:45:00.274781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.274744 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:00.333237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.324859 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.333237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.325499 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:00.336021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.335986 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.336590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.336561 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.336791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.336763 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.337068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.337049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4krx4\" (UniqueName: \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.382465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.381958 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:45:00.401909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.363936 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:45:00.448745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.448709 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:00.463457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.463243 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.464515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.464400 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.464686 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476465 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476562 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqz9\" (UniqueName: \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476629 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476702 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4zh\" (UniqueName: \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476796 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476833 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.476948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4krx4\" (UniqueName: \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.479475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.477014 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.480126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.479540 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.482268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.482238 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.483613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.483588 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.483912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.483898 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:00.493168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.490665 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:45:00.499922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.497209 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:00.505078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.504991 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:00.511612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.511578 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:00.524817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.524592 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:00.524817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.524711 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.534880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.534849 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:00.535236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.534980 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.540361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.540330 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4krx4\" (UniqueName: \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\") pod \"payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.541119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.541086 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\") pod \"payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.548685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.548590 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:00.556508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.554609 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:00.573298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.573267 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:00.573655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.573637 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.578333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.577923 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579428 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqz9\" (UniqueName: \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579554 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579600 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579641 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579667 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.579830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.579692 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs8hr\" (UniqueName: \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.580884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.580853 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9s9\" (UniqueName: \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.581004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.580972 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhjp\" (UniqueName: \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.581068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581006 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwzs\" (UniqueName: \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.581068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4zh\" (UniqueName: \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.581596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581286 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.581596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581338 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6tqv\" (UniqueName: \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.581596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581379 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.581596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.581501 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.582522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.582502 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.582679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.582533 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.614043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.614001 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqz9\" (UniqueName: \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\") pod \"maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.617366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.617329 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4zh\" (UniqueName: \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.682851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.682813 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.683045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.682875 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6tqv\" (UniqueName: \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.683045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.682963 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.683045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683041 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.683231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683077 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.683231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683112 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.683231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683137 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs8hr\" (UniqueName: \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.683231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683187 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9s9\" (UniqueName: \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.683231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683217 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhjp\" (UniqueName: \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.683560 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683245 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwzs\" (UniqueName: \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.683638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683607 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.683969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.683945 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.684282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.684259 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.684378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.684306 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.684557 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.684537 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.707366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.707325 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6tqv\" (UniqueName: \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.707366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.707359 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwzs\" (UniqueName: \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\") pod \"maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.711929 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.711902 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:00.720943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.720909 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9s9\" (UniqueName: \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\") pod \"maas-api-key-cleanup-29895405-9tcdg\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.729183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.729150 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs8hr\" (UniqueName: \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\") pod \"maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.729782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.729759 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhjp\" (UniqueName: \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\") pod \"maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:00.756946 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.756896 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:00.807822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.807034 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8943f3b0-aaac-41d7-bedc-e4da89f59359" path="/var/lib/kubelet/pods/8943f3b0-aaac-41d7-bedc-e4da89f59359/volumes" Nov 03 16:45:00.808473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.808456 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d335c03e-e302-4fba-b0dc-a00f46190420" path="/var/lib/kubelet/pods/d335c03e-e302-4fba-b0dc-a00f46190420/volumes" Nov 03 16:45:00.826495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.826261 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:00.832730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.832700 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:00.883123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.883044 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:00.919570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.918769 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:00.954225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.953975 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:00.970738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.970619 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:00.973636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:00.973423 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:01.041792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.041762 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:01.044320 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.044109 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89fd723_b89a_439a_86e8_21765b379933.slice/crio-bfc5a723307597ec3ef5fd2709d2e35f7ff4e953571a498ed983e442ebd467ca WatchSource:0}: Error finding container bfc5a723307597ec3ef5fd2709d2e35f7ff4e953571a498ed983e442ebd467ca: Status 404 returned error can't find the container with id bfc5a723307597ec3ef5fd2709d2e35f7ff4e953571a498ed983e442ebd467ca Nov 03 16:45:01.115615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.115545 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:45:01.234492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.234432 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" event={"ID":"e12a7cae-1690-4e5c-90aa-5a754e51302d","Type":"ContainerStarted","Data":"2f797756a6596dd2af755e15940f0e10431e2c44e834062327f9f978d172d893"} Nov 03 16:45:01.236566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.236532 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerStarted","Data":"bfc5a723307597ec3ef5fd2709d2e35f7ff4e953571a498ed983e442ebd467ca"} Nov 03 16:45:01.405833 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.405803 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24495681_148a_4354_91f1_26089a7b44b8.slice/crio-cf840481dbb804a6188e950c1ef7e0af76b0ee250699515b252595104f6435ea WatchSource:0}: Error finding container cf840481dbb804a6188e950c1ef7e0af76b0ee250699515b252595104f6435ea: Status 404 returned error can't find the container with id cf840481dbb804a6188e950c1ef7e0af76b0ee250699515b252595104f6435ea Nov 03 16:45:01.408365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.408341 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:01.409141 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.409118 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5685af88_fea8_45af_892d_5c08b750edc0.slice/crio-843fa5a54fe44f68988d40659675807f490d85623fec21dbb82e7845a0b7414f WatchSource:0}: Error finding container 843fa5a54fe44f68988d40659675807f490d85623fec21dbb82e7845a0b7414f: Status 404 returned error can't find the container with id 843fa5a54fe44f68988d40659675807f490d85623fec21dbb82e7845a0b7414f Nov 03 16:45:01.415109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.415083 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:45:01.419828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.419798 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:01.421682 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.421652 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87010ed2_0812_491f_ae0a_d39bca083d10.slice/crio-4c322d12ef838199b4bd3cf553614c23acfcc4650b746d7708f574dbefc4e35b WatchSource:0}: Error finding container 4c322d12ef838199b4bd3cf553614c23acfcc4650b746d7708f574dbefc4e35b: Status 404 returned error can't find the container with id 4c322d12ef838199b4bd3cf553614c23acfcc4650b746d7708f574dbefc4e35b Nov 03 16:45:01.449594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.449570 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:01.454022 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.453992 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod709973e0_ca0e_40b6_87a6_af1168e56849.slice/crio-491ab91d828b4386513f649fe88156d01e73e314ce83d2088d560115fc5c8f2d WatchSource:0}: Error finding container 491ab91d828b4386513f649fe88156d01e73e314ce83d2088d560115fc5c8f2d: Status 404 returned error can't find the container with id 491ab91d828b4386513f649fe88156d01e73e314ce83d2088d560115fc5c8f2d Nov 03 16:45:01.457766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.457733 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:01.496807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.496774 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:01.497436 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.497407 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9535c1b3_a8f4_4aca_bded_e847da45c8fa.slice/crio-eae8941b3dae41836976506e09bbf8ec2eaf4623e98dff41747bab214ec18782 WatchSource:0}: Error finding container eae8941b3dae41836976506e09bbf8ec2eaf4623e98dff41747bab214ec18782: Status 404 returned error can't find the container with id eae8941b3dae41836976506e09bbf8ec2eaf4623e98dff41747bab214ec18782 Nov 03 16:45:01.516837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:01.516807 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:01.517587 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:01.517562 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e755d4_267b_4203_82b6_67a120f3d3dd.slice/crio-891d98751cc047f5ae886bbc745599151b0b3b603058d9dd50b461e7d026c8a9 WatchSource:0}: Error finding container 891d98751cc047f5ae886bbc745599151b0b3b603058d9dd50b461e7d026c8a9: Status 404 returned error can't find the container with id 891d98751cc047f5ae886bbc745599151b0b3b603058d9dd50b461e7d026c8a9 Nov 03 16:45:02.251886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.251849 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerStarted","Data":"8a6313d5c206028a62ede785c2b3af1c02450f8a0ebbbbe71c5b0c3adde5da3d"} Nov 03 16:45:02.252312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.251922 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerStarted","Data":"491ab91d828b4386513f649fe88156d01e73e314ce83d2088d560115fc5c8f2d"} Nov 03 16:45:02.255389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.255362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerStarted","Data":"663da77483c2a8ddde54c959466e75bc5f982e033504c9f59764b9005fcbac8b"} Nov 03 16:45:02.255555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.255411 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerStarted","Data":"eae8941b3dae41836976506e09bbf8ec2eaf4623e98dff41747bab214ec18782"} Nov 03 16:45:02.258374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.258347 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerStarted","Data":"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e"} Nov 03 16:45:02.258709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.258646 6001 scope.go:122] "RemoveContainer" containerID="179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e" Nov 03 16:45:02.260531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.260435 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerStarted","Data":"581be7cf148a2da90d1fb867d17702a7e5e004e6f2771282cafec0d6a3eb6f95"} Nov 03 16:45:02.260531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.260495 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerStarted","Data":"4c322d12ef838199b4bd3cf553614c23acfcc4650b746d7708f574dbefc4e35b"} Nov 03 16:45:02.263242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.263165 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerStarted","Data":"e55be1b72a79e4613a191b1337d1c93718bdc37e5e66e32f62f176cf651c577f"} Nov 03 16:45:02.263242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.263210 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerStarted","Data":"9b17c2a31dc266533a90ff71ebb3bb85510a4e8a50542d17f4ece9f49a351a53"} Nov 03 16:45:02.265078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.265043 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" event={"ID":"5685af88-fea8-45af-892d-5c08b750edc0","Type":"ContainerStarted","Data":"9009104a84bbd349fffd2ea6ea3b7aa7e956749b8caa7fa1565bc664d8dc661f"} Nov 03 16:45:02.265160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.265104 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" event={"ID":"5685af88-fea8-45af-892d-5c08b750edc0","Type":"ContainerStarted","Data":"843fa5a54fe44f68988d40659675807f490d85623fec21dbb82e7845a0b7414f"} Nov 03 16:45:02.265160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.265136 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:02.267025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.267005 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" event={"ID":"e12a7cae-1690-4e5c-90aa-5a754e51302d","Type":"ContainerStarted","Data":"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea"} Nov 03 16:45:02.267115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.267079 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:02.268712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.268688 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerStarted","Data":"891d98751cc047f5ae886bbc745599151b0b3b603058d9dd50b461e7d026c8a9"} Nov 03 16:45:02.270332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.270313 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerStarted","Data":"a7d60d44194904f6e5152f3cce21d3f2edbb74e683f59b5e782c83124179f2ae"} Nov 03 16:45:02.270413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.270351 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerStarted","Data":"cf840481dbb804a6188e950c1ef7e0af76b0ee250699515b252595104f6435ea"} Nov 03 16:45:02.305853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.305802 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" podStartSLOduration=2.305789118 podStartE2EDuration="2.305789118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.301881008 +0000 UTC m=+1797.635094579" watchObservedRunningTime="2026-11-03 16:45:02.305789118 +0000 UTC m=+1797.639002727" Nov 03 16:45:02.415499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.415403 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" podStartSLOduration=2.415385523 podStartE2EDuration="2.415385523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.413168438 +0000 UTC m=+1797.746382008" watchObservedRunningTime="2026-11-03 16:45:02.415385523 +0000 UTC m=+1797.748599101" Nov 03 16:45:02.416499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.416409 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" podStartSLOduration=2.416390025 podStartE2EDuration="2.416390025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.361100162 +0000 UTC m=+1797.694313735" watchObservedRunningTime="2026-11-03 16:45:02.416390025 +0000 UTC m=+1797.749603651" Nov 03 16:45:02.463870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.463813 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" podStartSLOduration=2.463799709 podStartE2EDuration="2.463799709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.461556743 +0000 UTC m=+1797.794770316" watchObservedRunningTime="2026-11-03 16:45:02.463799709 +0000 UTC m=+1797.797013272" Nov 03 16:45:02.571679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.571628 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" podStartSLOduration=2.571613175 podStartE2EDuration="2.571613175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.569932752 +0000 UTC m=+1797.903146541" watchObservedRunningTime="2026-11-03 16:45:02.571613175 +0000 UTC m=+1797.904826743" Nov 03 16:45:02.616905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.616827 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" podStartSLOduration=2.61680389 podStartE2EDuration="2.61680389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.613544202 +0000 UTC m=+1797.946757771" watchObservedRunningTime="2026-11-03 16:45:02.61680389 +0000 UTC m=+1797.950017460" Nov 03 16:45:02.665626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.665496 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" podStartSLOduration=2.66548141 podStartE2EDuration="2.66548141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.659962545 +0000 UTC m=+1797.993176111" watchObservedRunningTime="2026-11-03 16:45:02.66548141 +0000 UTC m=+1797.998694978" Nov 03 16:45:02.708490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:02.708394 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" podStartSLOduration=2.708378423 podStartE2EDuration="2.708378423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:02.704603481 +0000 UTC m=+1798.037817052" watchObservedRunningTime="2026-11-03 16:45:02.708378423 +0000 UTC m=+1798.041592045" Nov 03 16:45:03.281355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.281321 6001 generic.go:356] "Generic (PLEG): container finished" podID="87010ed2-0812-491f-ae0a-d39bca083d10" containerID="581be7cf148a2da90d1fb867d17702a7e5e004e6f2771282cafec0d6a3eb6f95" exitCode=60 Nov 03 16:45:03.281792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.281408 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerDied","Data":"581be7cf148a2da90d1fb867d17702a7e5e004e6f2771282cafec0d6a3eb6f95"} Nov 03 16:45:03.281792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.281776 6001 scope.go:122] "RemoveContainer" containerID="581be7cf148a2da90d1fb867d17702a7e5e004e6f2771282cafec0d6a3eb6f95" Nov 03 16:45:03.283315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.283285 6001 generic.go:356] "Generic (PLEG): container finished" podID="a08c1da5-90ea-43ed-b917-318f687efc81" containerID="e55be1b72a79e4613a191b1337d1c93718bdc37e5e66e32f62f176cf651c577f" exitCode=60 Nov 03 16:45:03.283429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.283410 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerDied","Data":"e55be1b72a79e4613a191b1337d1c93718bdc37e5e66e32f62f176cf651c577f"} Nov 03 16:45:03.283796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.283742 6001 scope.go:122] "RemoveContainer" containerID="e55be1b72a79e4613a191b1337d1c93718bdc37e5e66e32f62f176cf651c577f" Nov 03 16:45:03.285429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.285406 6001 generic.go:356] "Generic (PLEG): container finished" podID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerID="ac842786617d890a5451fcf744df156b75235425774343b9f9fdbbef38645843" exitCode=60 Nov 03 16:45:03.285591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.285505 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerDied","Data":"ac842786617d890a5451fcf744df156b75235425774343b9f9fdbbef38645843"} Nov 03 16:45:03.285926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.285830 6001 scope.go:122] "RemoveContainer" containerID="ac842786617d890a5451fcf744df156b75235425774343b9f9fdbbef38645843" Nov 03 16:45:03.287409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.287391 6001 generic.go:356] "Generic (PLEG): container finished" podID="24495681-148a-4354-91f1-26089a7b44b8" containerID="a7d60d44194904f6e5152f3cce21d3f2edbb74e683f59b5e782c83124179f2ae" exitCode=60 Nov 03 16:45:03.287520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.287498 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerDied","Data":"a7d60d44194904f6e5152f3cce21d3f2edbb74e683f59b5e782c83124179f2ae"} Nov 03 16:45:03.287717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.287704 6001 scope.go:122] "RemoveContainer" containerID="a7d60d44194904f6e5152f3cce21d3f2edbb74e683f59b5e782c83124179f2ae" Nov 03 16:45:03.289294 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.289271 6001 generic.go:356] "Generic (PLEG): container finished" podID="709973e0-ca0e-40b6-87a6-af1168e56849" containerID="8a6313d5c206028a62ede785c2b3af1c02450f8a0ebbbbe71c5b0c3adde5da3d" exitCode=60 Nov 03 16:45:03.289408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.289359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerDied","Data":"8a6313d5c206028a62ede785c2b3af1c02450f8a0ebbbbe71c5b0c3adde5da3d"} Nov 03 16:45:03.289606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.289579 6001 scope.go:122] "RemoveContainer" containerID="8a6313d5c206028a62ede785c2b3af1c02450f8a0ebbbbe71c5b0c3adde5da3d" Nov 03 16:45:03.291094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.291076 6001 generic.go:356] "Generic (PLEG): container finished" podID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerID="663da77483c2a8ddde54c959466e75bc5f982e033504c9f59764b9005fcbac8b" exitCode=60 Nov 03 16:45:03.291202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.291143 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerDied","Data":"663da77483c2a8ddde54c959466e75bc5f982e033504c9f59764b9005fcbac8b"} Nov 03 16:45:03.291343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.291329 6001 scope.go:122] "RemoveContainer" containerID="663da77483c2a8ddde54c959466e75bc5f982e033504c9f59764b9005fcbac8b" Nov 03 16:45:03.293050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293028 6001 generic.go:356] "Generic (PLEG): container finished" podID="f89fd723-b89a-439a-86e8-21765b379933" containerID="179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e" exitCode=60 Nov 03 16:45:03.293050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293049 6001 generic.go:356] "Generic (PLEG): container finished" podID="f89fd723-b89a-439a-86e8-21765b379933" containerID="7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb" exitCode=60 Nov 03 16:45:03.293208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293108 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerDied","Data":"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e"} Nov 03 16:45:03.293208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293154 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerDied","Data":"7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb"} Nov 03 16:45:03.293208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293184 6001 scope.go:122] "RemoveContainer" containerID="179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e" Nov 03 16:45:03.293420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.293398 6001 scope.go:122] "RemoveContainer" containerID="7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb" Nov 03 16:45:03.293706 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:03.293682 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw_odh-ai-gateway-infra(f89fd723-b89a-439a-86e8-21765b379933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" podUID="f89fd723-b89a-439a-86e8-21765b379933" Nov 03 16:45:03.311769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.311745 6001 scope.go:122] "RemoveContainer" containerID="179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e" Nov 03 16:45:03.312101 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:03.312079 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e\": container with ID starting with 179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e not found: ID does not exist" containerID="179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e" Nov 03 16:45:03.312180 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:03.312114 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e"} err="failed to get container status \"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e\": rpc error: code = NotFound desc = could not find container \"179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e\": container with ID starting with 179429a0a866771d6507448246cc1780fae955f706c4250db7d108d293d41a9e not found: ID does not exist" Nov 03 16:45:04.304348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.304304 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerStarted","Data":"a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e"} Nov 03 16:45:04.306537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.306502 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerStarted","Data":"be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739"} Nov 03 16:45:04.308637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.308612 6001 scope.go:122] "RemoveContainer" containerID="7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb" Nov 03 16:45:04.309045 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:04.309012 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw_odh-ai-gateway-infra(f89fd723-b89a-439a-86e8-21765b379933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" podUID="f89fd723-b89a-439a-86e8-21765b379933" Nov 03 16:45:04.310391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.310363 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerStarted","Data":"f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490"} Nov 03 16:45:04.312528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.312500 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerStarted","Data":"b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd"} Nov 03 16:45:04.314682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:04.314661 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerStarted","Data":"fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4"} Nov 03 16:45:05.060615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.060533 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:45:05.071096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.071069 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:45:05.329731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.329621 6001 generic.go:356] "Generic (PLEG): container finished" podID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerID="06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724" exitCode=60 Nov 03 16:45:05.330185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.330016 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerDied","Data":"06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724"} Nov 03 16:45:05.330185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.330099 6001 scope.go:122] "RemoveContainer" containerID="663da77483c2a8ddde54c959466e75bc5f982e033504c9f59764b9005fcbac8b" Nov 03 16:45:05.330309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.330268 6001 scope.go:122] "RemoveContainer" containerID="06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724" Nov 03 16:45:05.330744 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.330713 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq_odh-ai-gateway-infra(9535c1b3-a8f4-4aca-bded-e847da45c8fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" Nov 03 16:45:05.332401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.332372 6001 generic.go:356] "Generic (PLEG): container finished" podID="87010ed2-0812-491f-ae0a-d39bca083d10" containerID="f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490" exitCode=60 Nov 03 16:45:05.332516 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.332467 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerDied","Data":"f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490"} Nov 03 16:45:05.332764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.332747 6001 scope.go:122] "RemoveContainer" containerID="f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490" Nov 03 16:45:05.333148 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.333125 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls_odh-ai-gateway-infra(87010ed2-0812-491f-ae0a-d39bca083d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" Nov 03 16:45:05.334490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.334431 6001 generic.go:356] "Generic (PLEG): container finished" podID="a08c1da5-90ea-43ed-b917-318f687efc81" containerID="b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd" exitCode=60 Nov 03 16:45:05.334599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.334473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerDied","Data":"b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd"} Nov 03 16:45:05.334793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.334778 6001 scope.go:122] "RemoveContainer" containerID="b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd" Nov 03 16:45:05.335105 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.335083 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv_odh-ai-gateway-infra(a08c1da5-90ea-43ed-b917-318f687efc81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" Nov 03 16:45:05.336685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.336663 6001 generic.go:356] "Generic (PLEG): container finished" podID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerID="fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4" exitCode=60 Nov 03 16:45:05.336761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.336733 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerDied","Data":"fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4"} Nov 03 16:45:05.336923 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.336910 6001 scope.go:122] "RemoveContainer" containerID="fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4" Nov 03 16:45:05.337202 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.337183 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-9tcdg_odh-ai-gateway-infra(68e755d4-267b-4203-82b6-67a120f3d3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" Nov 03 16:45:05.338907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.338886 6001 generic.go:356] "Generic (PLEG): container finished" podID="24495681-148a-4354-91f1-26089a7b44b8" containerID="a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e" exitCode=60 Nov 03 16:45:05.339009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.338951 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerDied","Data":"a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e"} Nov 03 16:45:05.339252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.339232 6001 scope.go:122] "RemoveContainer" containerID="a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e" Nov 03 16:45:05.339561 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.339538 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g_odh-ai-gateway-infra(24495681-148a-4354-91f1-26089a7b44b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" podUID="24495681-148a-4354-91f1-26089a7b44b8" Nov 03 16:45:05.341021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.340999 6001 generic.go:356] "Generic (PLEG): container finished" podID="709973e0-ca0e-40b6-87a6-af1168e56849" containerID="be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739" exitCode=60 Nov 03 16:45:05.341112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.341088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerDied","Data":"be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739"} Nov 03 16:45:05.341256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.341243 6001 scope.go:122] "RemoveContainer" containerID="be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739" Nov 03 16:45:05.341565 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:05.341541 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng_odh-ai-gateway-infra(709973e0-ca0e-40b6-87a6-af1168e56849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" Nov 03 16:45:05.345708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.345689 6001 scope.go:122] "RemoveContainer" containerID="581be7cf148a2da90d1fb867d17702a7e5e004e6f2771282cafec0d6a3eb6f95" Nov 03 16:45:05.360967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.360781 6001 scope.go:122] "RemoveContainer" containerID="e55be1b72a79e4613a191b1337d1c93718bdc37e5e66e32f62f176cf651c577f" Nov 03 16:45:05.376882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.376862 6001 scope.go:122] "RemoveContainer" containerID="ac842786617d890a5451fcf744df156b75235425774343b9f9fdbbef38645843" Nov 03 16:45:05.391889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.391863 6001 scope.go:122] "RemoveContainer" containerID="a7d60d44194904f6e5152f3cce21d3f2edbb74e683f59b5e782c83124179f2ae" Nov 03 16:45:05.406825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:05.406805 6001 scope.go:122] "RemoveContainer" containerID="8a6313d5c206028a62ede785c2b3af1c02450f8a0ebbbbe71c5b0c3adde5da3d" Nov 03 16:45:06.352568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.352536 6001 scope.go:122] "RemoveContainer" containerID="06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724" Nov 03 16:45:06.353058 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.352997 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq_odh-ai-gateway-infra(9535c1b3-a8f4-4aca-bded-e847da45c8fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" Nov 03 16:45:06.354381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.354351 6001 scope.go:122] "RemoveContainer" containerID="f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490" Nov 03 16:45:06.354764 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.354740 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls_odh-ai-gateway-infra(87010ed2-0812-491f-ae0a-d39bca083d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" Nov 03 16:45:06.356232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.356204 6001 scope.go:122] "RemoveContainer" containerID="b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd" Nov 03 16:45:06.356626 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.356598 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv_odh-ai-gateway-infra(a08c1da5-90ea-43ed-b917-318f687efc81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" Nov 03 16:45:06.358099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.358074 6001 scope.go:122] "RemoveContainer" containerID="fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4" Nov 03 16:45:06.358419 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.358396 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-9tcdg_odh-ai-gateway-infra(68e755d4-267b-4203-82b6-67a120f3d3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" Nov 03 16:45:06.359944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.359925 6001 scope.go:122] "RemoveContainer" containerID="a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e" Nov 03 16:45:06.360240 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.360215 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g_odh-ai-gateway-infra(24495681-148a-4354-91f1-26089a7b44b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" podUID="24495681-148a-4354-91f1-26089a7b44b8" Nov 03 16:45:06.361791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:06.361773 6001 scope.go:122] "RemoveContainer" containerID="be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739" Nov 03 16:45:06.362027 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:06.362010 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng_odh-ai-gateway-infra(709973e0-ca0e-40b6-87a6-af1168e56849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" Nov 03 16:45:08.269633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:08.269598 6001 scope.go:122] "RemoveContainer" containerID="b7b8e109e93ffa436f22cc3d35fe91b9adb4aa5c4cec83fae72d5eecd60ba0fe" Nov 03 16:45:08.281455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:08.281417 6001 scope.go:122] "RemoveContainer" containerID="ec8d60cba2afa0547b04c031839ed8fdf55579f0888cf9352007dea56c47c44a" Nov 03 16:45:08.295523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:08.295497 6001 scope.go:122] "RemoveContainer" containerID="86971b05269bf5201af64fde131ddaf2608b3eb93272eda1e34d4badb4f38e6e" Nov 03 16:45:09.059003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:09.058972 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9"] Nov 03 16:45:09.079490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:09.079397 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43033229-sv4t9"] Nov 03 16:45:09.211872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:09.211844 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:45:10.802861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:10.802825 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371926e7-0985-4f2e-9acf-8eb84ea521ce" path="/var/lib/kubelet/pods/371926e7-0985-4f2e-9acf-8eb84ea521ce/volumes" Nov 03 16:45:13.296418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:13.296376 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:45:13.296955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:13.296513 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:45:14.789414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:14.789374 6001 scope.go:122] "RemoveContainer" containerID="a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e" Nov 03 16:45:14.789910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:14.789661 6001 scope.go:122] "RemoveContainer" containerID="7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb" Nov 03 16:45:14.789910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:14.789695 6001 scope.go:122] "RemoveContainer" containerID="f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490" Nov 03 16:45:15.528531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.528494 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerStarted","Data":"82eee204a8f4f1f44ed529bb6ff5ba3e0890ef826e45cdc3420e158fa0e17435"} Nov 03 16:45:15.530698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.530661 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerStarted","Data":"6b06c586b55e760aded4d8fddcda202fa0dbc577a2f95d77b467ff6c70cc2f08"} Nov 03 16:45:15.532806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.532777 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerStarted","Data":"d66039b1387457365aeb4c2e5db67a85827abe9f07a687cd13c2c3c70136801f"} Nov 03 16:45:15.660140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.660016 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" podStartSLOduration=15.659998922 podStartE2EDuration="15.659998922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:15.65577032 +0000 UTC m=+1810.988983908" watchObservedRunningTime="2026-11-03 16:45:15.659998922 +0000 UTC m=+1810.993212491" Nov 03 16:45:15.781884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.781850 6001 scope.go:122] "RemoveContainer" containerID="b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd" Nov 03 16:45:15.782181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.782161 6001 scope.go:122] "RemoveContainer" containerID="be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739" Nov 03 16:45:15.782318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:15.782301 6001 scope.go:122] "RemoveContainer" containerID="fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4" Nov 03 16:45:16.544302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.544208 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerStarted","Data":"9c50baed59eff9b051c3276cd1ce4d2b4b8443ae250fe6cce69a4fbcf1d1d88e"} Nov 03 16:45:16.547773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.547741 6001 generic.go:356] "Generic (PLEG): container finished" podID="f89fd723-b89a-439a-86e8-21765b379933" containerID="d66039b1387457365aeb4c2e5db67a85827abe9f07a687cd13c2c3c70136801f" exitCode=60 Nov 03 16:45:16.547928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.547808 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerDied","Data":"d66039b1387457365aeb4c2e5db67a85827abe9f07a687cd13c2c3c70136801f"} Nov 03 16:45:16.547928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.547871 6001 scope.go:122] "RemoveContainer" containerID="7eef97ede77ea1e441e3c6290427334d23b9e6b7428434698b10bca6ffe80aeb" Nov 03 16:45:16.548410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.548374 6001 scope.go:122] "RemoveContainer" containerID="d66039b1387457365aeb4c2e5db67a85827abe9f07a687cd13c2c3c70136801f" Nov 03 16:45:16.548829 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:16.548807 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw_odh-ai-gateway-infra(f89fd723-b89a-439a-86e8-21765b379933)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" podUID="f89fd723-b89a-439a-86e8-21765b379933" Nov 03 16:45:16.550496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.550474 6001 generic.go:356] "Generic (PLEG): container finished" podID="87010ed2-0812-491f-ae0a-d39bca083d10" containerID="82eee204a8f4f1f44ed529bb6ff5ba3e0890ef826e45cdc3420e158fa0e17435" exitCode=60 Nov 03 16:45:16.550602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.550579 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerDied","Data":"82eee204a8f4f1f44ed529bb6ff5ba3e0890ef826e45cdc3420e158fa0e17435"} Nov 03 16:45:16.551057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.551039 6001 scope.go:122] "RemoveContainer" containerID="82eee204a8f4f1f44ed529bb6ff5ba3e0890ef826e45cdc3420e158fa0e17435" Nov 03 16:45:16.551502 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:16.551477 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls_odh-ai-gateway-infra(87010ed2-0812-491f-ae0a-d39bca083d10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" Nov 03 16:45:16.552850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.552816 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerStarted","Data":"0fcba3a2290cba85f196144af84d428f03893ac70bde6e956c6601d53b0abec5"} Nov 03 16:45:16.556892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.556865 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerStarted","Data":"944cd0666291aed927a89695d709b7ae14ff2dbb3d541003466e90a248bd2925"} Nov 03 16:45:16.560300 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.560274 6001 generic.go:356] "Generic (PLEG): container finished" podID="24495681-148a-4354-91f1-26089a7b44b8" containerID="6b06c586b55e760aded4d8fddcda202fa0dbc577a2f95d77b467ff6c70cc2f08" exitCode=60 Nov 03 16:45:16.560417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.560327 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerDied","Data":"6b06c586b55e760aded4d8fddcda202fa0dbc577a2f95d77b467ff6c70cc2f08"} Nov 03 16:45:16.560795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.560773 6001 scope.go:122] "RemoveContainer" containerID="6b06c586b55e760aded4d8fddcda202fa0dbc577a2f95d77b467ff6c70cc2f08" Nov 03 16:45:16.561260 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:16.561224 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g_odh-ai-gateway-infra(24495681-148a-4354-91f1-26089a7b44b8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" podUID="24495681-148a-4354-91f1-26089a7b44b8" Nov 03 16:45:16.565692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.565615 6001 scope.go:122] "RemoveContainer" containerID="f0d85067ce67a6400dba8e670ebd78cab69f8e2e256e76600ffd441ddc0b6490" Nov 03 16:45:16.583473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.583400 6001 scope.go:122] "RemoveContainer" containerID="a921e8bc8bb7eb79e491560f42e3220b60debd9baf1e7ff845e8a13ef336355e" Nov 03 16:45:16.607341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.607301 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:16.645372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.645332 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:16.677348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:16.677308 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:17.044060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.044020 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4"] Nov 03 16:45:17.049821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.049787 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba401d0-jf9v4"] Nov 03 16:45:17.571471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.571408 6001 generic.go:356] "Generic (PLEG): container finished" podID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerID="944cd0666291aed927a89695d709b7ae14ff2dbb3d541003466e90a248bd2925" exitCode=60 Nov 03 16:45:17.571966 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.571498 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerDied","Data":"944cd0666291aed927a89695d709b7ae14ff2dbb3d541003466e90a248bd2925"} Nov 03 16:45:17.571966 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.571562 6001 scope.go:122] "RemoveContainer" containerID="fff616fbdb57d9f969c595827d0af16751ce619eea45c3751e158d83f3e657b4" Nov 03 16:45:17.572067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.572012 6001 scope.go:122] "RemoveContainer" containerID="944cd0666291aed927a89695d709b7ae14ff2dbb3d541003466e90a248bd2925" Nov 03 16:45:17.572501 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:17.572476 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895405-9tcdg_odh-ai-gateway-infra(68e755d4-267b-4203-82b6-67a120f3d3dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" Nov 03 16:45:17.575805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.575785 6001 generic.go:356] "Generic (PLEG): container finished" podID="709973e0-ca0e-40b6-87a6-af1168e56849" containerID="9c50baed59eff9b051c3276cd1ce4d2b4b8443ae250fe6cce69a4fbcf1d1d88e" exitCode=60 Nov 03 16:45:17.575918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.575868 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerDied","Data":"9c50baed59eff9b051c3276cd1ce4d2b4b8443ae250fe6cce69a4fbcf1d1d88e"} Nov 03 16:45:17.576364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.576340 6001 scope.go:122] "RemoveContainer" containerID="9c50baed59eff9b051c3276cd1ce4d2b4b8443ae250fe6cce69a4fbcf1d1d88e" Nov 03 16:45:17.576800 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:17.576771 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng_odh-ai-gateway-infra(709973e0-ca0e-40b6-87a6-af1168e56849)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" Nov 03 16:45:17.582102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.582082 6001 generic.go:356] "Generic (PLEG): container finished" podID="a08c1da5-90ea-43ed-b917-318f687efc81" containerID="0fcba3a2290cba85f196144af84d428f03893ac70bde6e956c6601d53b0abec5" exitCode=60 Nov 03 16:45:17.582201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.582118 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerDied","Data":"0fcba3a2290cba85f196144af84d428f03893ac70bde6e956c6601d53b0abec5"} Nov 03 16:45:17.582540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.582522 6001 scope.go:122] "RemoveContainer" containerID="0fcba3a2290cba85f196144af84d428f03893ac70bde6e956c6601d53b0abec5" Nov 03 16:45:17.582931 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:17.582905 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv_odh-ai-gateway-infra(a08c1da5-90ea-43ed-b917-318f687efc81)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" Nov 03 16:45:17.587780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.587737 6001 scope.go:122] "RemoveContainer" containerID="be717313bf10aed3c9ce7e479a390e65cee8caff218635e56197a02fdfae6739" Nov 03 16:45:17.618378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.618225 6001 scope.go:122] "RemoveContainer" containerID="b6a5c45256cbef3c6729ce832ad41a3eace1b2bcd5200df7cc4fb7f1abdceffd" Nov 03 16:45:17.652503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.651464 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:17.693317 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.686335 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:17.821652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.821566 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:17.893763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.893736 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:17.898570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.898539 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:17.933136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.933096 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\") pod \"24495681-148a-4354-91f1-26089a7b44b8\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " Nov 03 16:45:17.933549 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.933494 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\" (UniqueName: \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\") pod \"24495681-148a-4354-91f1-26089a7b44b8\" (UID: \"24495681-148a-4354-91f1-26089a7b44b8\") " Nov 03 16:45:17.933748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.933556 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca" pod "24495681-148a-4354-91f1-26089a7b44b8" (UID: "24495681-148a-4354-91f1-26089a7b44b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:17.934295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.934276 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24495681-148a-4354-91f1-26089a7b44b8-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:17.935936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.935909 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr" pod "24495681-148a-4354-91f1-26089a7b44b8" (UID: "24495681-148a-4354-91f1-26089a7b44b8"). InnerVolumeSpecName "kube-api-access-gs8hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:17.981514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:17.981477 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:18.035500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.035431 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\" (UniqueName: \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\") pod \"f89fd723-b89a-439a-86e8-21765b379933\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " Nov 03 16:45:18.035671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.035580 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\") pod \"87010ed2-0812-491f-ae0a-d39bca083d10\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " Nov 03 16:45:18.035671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.035661 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\") pod \"f89fd723-b89a-439a-86e8-21765b379933\" (UID: \"f89fd723-b89a-439a-86e8-21765b379933\") " Nov 03 16:45:18.035865 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.035843 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\" (UniqueName: \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\") pod \"87010ed2-0812-491f-ae0a-d39bca083d10\" (UID: \"87010ed2-0812-491f-ae0a-d39bca083d10\") " Nov 03 16:45:18.035981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.035952 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca" pod "87010ed2-0812-491f-ae0a-d39bca083d10" (UID: "87010ed2-0812-491f-ae0a-d39bca083d10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:18.036102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.036041 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca" pod "f89fd723-b89a-439a-86e8-21765b379933" (UID: "f89fd723-b89a-439a-86e8-21765b379933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:18.036525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.036508 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89fd723-b89a-439a-86e8-21765b379933-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:18.036583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.036527 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs8hr\" (UniqueName: \"kubernetes.io/projected/24495681-148a-4354-91f1-26089a7b44b8-kube-api-access-gs8hr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:18.036583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.036536 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87010ed2-0812-491f-ae0a-d39bca083d10-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:18.037794 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.037766 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh" pod "f89fd723-b89a-439a-86e8-21765b379933" (UID: "f89fd723-b89a-439a-86e8-21765b379933"). InnerVolumeSpecName "kube-api-access-nm4zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:18.037892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.037874 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9" pod "87010ed2-0812-491f-ae0a-d39bca083d10" (UID: "87010ed2-0812-491f-ae0a-d39bca083d10"). InnerVolumeSpecName "kube-api-access-wqqz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:18.138998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.138896 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqz9\" (UniqueName: \"kubernetes.io/projected/87010ed2-0812-491f-ae0a-d39bca083d10-kube-api-access-wqqz9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:18.138998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.138938 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm4zh\" (UniqueName: \"kubernetes.io/projected/f89fd723-b89a-439a-86e8-21765b379933-kube-api-access-nm4zh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:18.591336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.591304 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" Nov 03 16:45:18.591808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.591336 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw" event={"ID":"f89fd723-b89a-439a-86e8-21765b379933","Type":"ContainerDied","Data":"bfc5a723307597ec3ef5fd2709d2e35f7ff4e953571a498ed983e442ebd467ca"} Nov 03 16:45:18.591808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.591405 6001 scope.go:122] "RemoveContainer" containerID="d66039b1387457365aeb4c2e5db67a85827abe9f07a687cd13c2c3c70136801f" Nov 03 16:45:18.593148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.593121 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" Nov 03 16:45:18.593238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.593156 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls" event={"ID":"87010ed2-0812-491f-ae0a-d39bca083d10","Type":"ContainerDied","Data":"4c322d12ef838199b4bd3cf553614c23acfcc4650b746d7708f574dbefc4e35b"} Nov 03 16:45:18.601127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.601101 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" Nov 03 16:45:18.601247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.601150 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g" event={"ID":"24495681-148a-4354-91f1-26089a7b44b8","Type":"ContainerDied","Data":"cf840481dbb804a6188e950c1ef7e0af76b0ee250699515b252595104f6435ea"} Nov 03 16:45:18.606845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.606647 6001 scope.go:122] "RemoveContainer" containerID="82eee204a8f4f1f44ed529bb6ff5ba3e0890ef826e45cdc3420e158fa0e17435" Nov 03 16:45:18.619325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.619295 6001 scope.go:122] "RemoveContainer" containerID="6b06c586b55e760aded4d8fddcda202fa0dbc577a2f95d77b467ff6c70cc2f08" Nov 03 16:45:18.651767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.651732 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:18.659162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.659129 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895405-fnmsw"] Nov 03 16:45:18.816640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.816602 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4d706a-281c-4f9e-9faf-081b4e8252e6" path="/var/lib/kubelet/pods/ad4d706a-281c-4f9e-9faf-081b4e8252e6/volumes" Nov 03 16:45:18.817335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.817313 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89fd723-b89a-439a-86e8-21765b379933" path="/var/lib/kubelet/pods/f89fd723-b89a-439a-86e8-21765b379933/volumes" Nov 03 16:45:18.817847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.817827 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:18.817915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.817872 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-db3c6b-29895405-p6fls"] Nov 03 16:45:18.910459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.910405 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:18.913714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.913685 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:18.916662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.916627 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-8650a6e8-29895405-mkz6g"] Nov 03 16:45:18.958733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.958581 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:18.976867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:18.976832 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:19.058370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.058330 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\" (UniqueName: \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\") pod \"68e755d4-267b-4203-82b6-67a120f3d3dd\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " Nov 03 16:45:19.058614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.058545 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\") pod \"a08c1da5-90ea-43ed-b917-318f687efc81\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " Nov 03 16:45:19.058736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.058713 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\" (UniqueName: \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\") pod \"a08c1da5-90ea-43ed-b917-318f687efc81\" (UID: \"a08c1da5-90ea-43ed-b917-318f687efc81\") " Nov 03 16:45:19.058889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.058791 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\") pod \"68e755d4-267b-4203-82b6-67a120f3d3dd\" (UID: \"68e755d4-267b-4203-82b6-67a120f3d3dd\") " Nov 03 16:45:19.058976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.058947 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca" pod "a08c1da5-90ea-43ed-b917-318f687efc81" (UID: "a08c1da5-90ea-43ed-b917-318f687efc81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.059411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.059384 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca" pod "68e755d4-267b-4203-82b6-67a120f3d3dd" (UID: "68e755d4-267b-4203-82b6-67a120f3d3dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.059679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.059656 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08c1da5-90ea-43ed-b917-318f687efc81-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.059756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.059684 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e755d4-267b-4203-82b6-67a120f3d3dd-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.061110 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.061077 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9" pod "68e755d4-267b-4203-82b6-67a120f3d3dd" (UID: "68e755d4-267b-4203-82b6-67a120f3d3dd"). InnerVolumeSpecName "kube-api-access-gn9s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.061233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.061214 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv" pod "a08c1da5-90ea-43ed-b917-318f687efc81" (UID: "a08c1da5-90ea-43ed-b917-318f687efc81"). InnerVolumeSpecName "kube-api-access-m6tqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.161024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.160934 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\") pod \"709973e0-ca0e-40b6-87a6-af1168e56849\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " Nov 03 16:45:19.161190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.161050 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\" (UniqueName: \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\") pod \"709973e0-ca0e-40b6-87a6-af1168e56849\" (UID: \"709973e0-ca0e-40b6-87a6-af1168e56849\") " Nov 03 16:45:19.161323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.161295 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca" pod "709973e0-ca0e-40b6-87a6-af1168e56849" (UID: "709973e0-ca0e-40b6-87a6-af1168e56849"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:19.161823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.161803 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6tqv\" (UniqueName: \"kubernetes.io/projected/a08c1da5-90ea-43ed-b917-318f687efc81-kube-api-access-m6tqv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.161888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.161824 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/709973e0-ca0e-40b6-87a6-af1168e56849-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.161888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.161839 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9s9\" (UniqueName: \"kubernetes.io/projected/68e755d4-267b-4203-82b6-67a120f3d3dd-kube-api-access-gn9s9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.163277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.163253 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp" pod "709973e0-ca0e-40b6-87a6-af1168e56849" (UID: "709973e0-ca0e-40b6-87a6-af1168e56849"). InnerVolumeSpecName "kube-api-access-frhjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:19.263129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.263089 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhjp\" (UniqueName: \"kubernetes.io/projected/709973e0-ca0e-40b6-87a6-af1168e56849-kube-api-access-frhjp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:19.352964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.352922 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:45:19.354162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354112 6001 cpu_manager.go:414] "containerMap: removing container" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354144 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354165 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354174 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354234 6001 cpu_manager.go:414] "containerMap: removing container" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354242 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354258 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354265 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354277 6001 cpu_manager.go:414] "containerMap: removing container" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354284 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354300 6001 cpu_manager.go:414] "containerMap: removing container" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354307 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354317 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354324 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354349 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354357 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354373 6001 cpu_manager.go:414] "containerMap: removing container" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354380 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354403 6001 cpu_manager.go:414] "containerMap: removing container" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354409 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354419 6001 cpu_manager.go:414] "containerMap: removing container" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354426 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354439 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354459 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354479 6001 cpu_manager.go:414] "containerMap: removing container" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354489 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354504 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354516 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.354511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354533 6001 cpu_manager.go:414] "containerMap: removing container" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354543 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354580 6001 cpu_manager.go:414] "containerMap: removing container" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354592 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354634 6001 cpu_manager.go:414] "containerMap: removing container" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354640 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354824 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354837 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354845 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354856 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354864 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354875 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354885 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354896 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354905 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354912 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354923 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354931 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354941 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.355416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.354952 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89fd723-b89a-439a-86e8-21765b379933" containerName="cleanup" Nov 03 16:45:19.359933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.359894 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.416563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.412355 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ldnz\" (UniqueName: \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.416563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.412619 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.416563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.413486 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-53bd50-dockercfg-v5brf\"" Nov 03 16:45:19.560873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.434688 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-53bd50\"" Nov 03 16:45:19.560873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.506509 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:45:19.560873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.519119 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.560873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.519311 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ldnz\" (UniqueName: \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.560873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.534777 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.568000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.567892 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ldnz\" (UniqueName: \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\") pod \"payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.624771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.624723 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" event={"ID":"a08c1da5-90ea-43ed-b917-318f687efc81","Type":"ContainerDied","Data":"9b17c2a31dc266533a90ff71ebb3bb85510a4e8a50542d17f4ece9f49a351a53"} Nov 03 16:45:19.625151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.624838 6001 scope.go:122] "RemoveContainer" containerID="0fcba3a2290cba85f196144af84d428f03893ac70bde6e956c6601d53b0abec5" Nov 03 16:45:19.625151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.624973 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv" Nov 03 16:45:19.633147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.633124 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" Nov 03 16:45:19.633342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.633112 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg" event={"ID":"68e755d4-267b-4203-82b6-67a120f3d3dd","Type":"ContainerDied","Data":"891d98751cc047f5ae886bbc745599151b0b3b603058d9dd50b461e7d026c8a9"} Nov 03 16:45:19.642170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.642145 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" Nov 03 16:45:19.643694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.642185 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng" event={"ID":"709973e0-ca0e-40b6-87a6-af1168e56849","Type":"ContainerDied","Data":"491ab91d828b4386513f649fe88156d01e73e314ce83d2088d560115fc5c8f2d"} Nov 03 16:45:19.648554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.648435 6001 scope.go:122] "RemoveContainer" containerID="944cd0666291aed927a89695d709b7ae14ff2dbb3d541003466e90a248bd2925" Nov 03 16:45:19.666407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.666384 6001 scope.go:122] "RemoveContainer" containerID="9c50baed59eff9b051c3276cd1ce4d2b4b8443ae250fe6cce69a4fbcf1d1d88e" Nov 03 16:45:19.671868 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:19.671834 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08c1da5_90ea_43ed_b917_318f687efc81.slice/crio-9b17c2a31dc266533a90ff71ebb3bb85510a4e8a50542d17f4ece9f49a351a53\": RecentStats: unable to find data in memory cache]" Nov 03 16:45:19.687676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.687640 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:19.692539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.692498 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895405-n6tcv"] Nov 03 16:45:19.747822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.747791 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:19.751837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.751815 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895405-9tcdg"] Nov 03 16:45:19.782710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.782679 6001 scope.go:122] "RemoveContainer" containerID="06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724" Nov 03 16:45:19.785335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.785316 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:19.812632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.812605 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:19.852426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:19.852391 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bffe15-29895405-5nmng"] Nov 03 16:45:20.107915 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:20.107874 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08df94b8_dca8_4f34_9b09_7f2d14804142.slice/crio-b36fe22fc8484e6ec6bde5d0e731a26b05c96ea7a7c40059d17ef80e228adee3 WatchSource:0}: Error finding container b36fe22fc8484e6ec6bde5d0e731a26b05c96ea7a7c40059d17ef80e228adee3: Status 404 returned error can't find the container with id b36fe22fc8484e6ec6bde5d0e731a26b05c96ea7a7c40059d17ef80e228adee3 Nov 03 16:45:20.109075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.109046 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:45:20.658092 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.658052 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" event={"ID":"08df94b8-dca8-4f34-9b09-7f2d14804142","Type":"ContainerStarted","Data":"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d"} Nov 03 16:45:20.658645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.658119 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" event={"ID":"08df94b8-dca8-4f34-9b09-7f2d14804142","Type":"ContainerStarted","Data":"b36fe22fc8484e6ec6bde5d0e731a26b05c96ea7a7c40059d17ef80e228adee3"} Nov 03 16:45:20.658645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.658151 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:20.661573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.661544 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerStarted","Data":"dc52611430a4ac7ce86595e766b323fdea9d6c825d4379d55cae478d7eddbf8d"} Nov 03 16:45:20.713109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.713042 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" podStartSLOduration=1.713026041 podStartE2EDuration="1.713026041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:20.70875935 +0000 UTC m=+1816.041972919" watchObservedRunningTime="2026-11-03 16:45:20.713026041 +0000 UTC m=+1816.046239609" Nov 03 16:45:20.792686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.792650 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24495681-148a-4354-91f1-26089a7b44b8" path="/var/lib/kubelet/pods/24495681-148a-4354-91f1-26089a7b44b8/volumes" Nov 03 16:45:20.793190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.793176 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" path="/var/lib/kubelet/pods/68e755d4-267b-4203-82b6-67a120f3d3dd/volumes" Nov 03 16:45:20.793700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.793683 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" path="/var/lib/kubelet/pods/709973e0-ca0e-40b6-87a6-af1168e56849/volumes" Nov 03 16:45:20.794452 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.794426 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" path="/var/lib/kubelet/pods/87010ed2-0812-491f-ae0a-d39bca083d10/volumes" Nov 03 16:45:20.794885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:20.794871 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" path="/var/lib/kubelet/pods/a08c1da5-90ea-43ed-b917-318f687efc81/volumes" Nov 03 16:45:21.671345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:21.671313 6001 generic.go:356] "Generic (PLEG): container finished" podID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerID="dc52611430a4ac7ce86595e766b323fdea9d6c825d4379d55cae478d7eddbf8d" exitCode=60 Nov 03 16:45:21.671887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:21.671388 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerDied","Data":"dc52611430a4ac7ce86595e766b323fdea9d6c825d4379d55cae478d7eddbf8d"} Nov 03 16:45:21.671887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:21.671481 6001 scope.go:122] "RemoveContainer" containerID="06e52c2c888e473420a2ca4904b2a068ff2580e65e41b08a201db696ffac8724" Nov 03 16:45:21.672017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:21.671952 6001 scope.go:122] "RemoveContainer" containerID="dc52611430a4ac7ce86595e766b323fdea9d6c825d4379d55cae478d7eddbf8d" Nov 03 16:45:21.672363 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:21.672339 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq_odh-ai-gateway-infra(9535c1b3-a8f4-4aca-bded-e847da45c8fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" Nov 03 16:45:21.778414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:21.778370 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:22.253541 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:22.253504 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:45:22.889702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.889672 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:22.980127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.980093 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\" (UniqueName: \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\") pod \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " Nov 03 16:45:22.980402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.980288 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\") pod \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\" (UID: \"9535c1b3-a8f4-4aca-bded-e847da45c8fa\") " Nov 03 16:45:22.981659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.981633 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca" pod "9535c1b3-a8f4-4aca-bded-e847da45c8fa" (UID: "9535c1b3-a8f4-4aca-bded-e847da45c8fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:22.983027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.982997 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9535c1b3-a8f4-4aca-bded-e847da45c8fa-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:22.983536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:22.983508 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs" pod "9535c1b3-a8f4-4aca-bded-e847da45c8fa" (UID: "9535c1b3-a8f4-4aca-bded-e847da45c8fa"). InnerVolumeSpecName "kube-api-access-8mwzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:23.084768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.084730 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mwzs\" (UniqueName: \"kubernetes.io/projected/9535c1b3-a8f4-4aca-bded-e847da45c8fa-kube-api-access-8mwzs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:23.696373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.696330 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" event={"ID":"9535c1b3-a8f4-4aca-bded-e847da45c8fa","Type":"ContainerDied","Data":"eae8941b3dae41836976506e09bbf8ec2eaf4623e98dff41747bab214ec18782"} Nov 03 16:45:23.696615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.696409 6001 scope.go:122] "RemoveContainer" containerID="dc52611430a4ac7ce86595e766b323fdea9d6c825d4379d55cae478d7eddbf8d" Nov 03 16:45:23.696615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.696566 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq" Nov 03 16:45:23.761508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.760159 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:23.768237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:23.768200 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-73cc79-29895405-9xqtq"] Nov 03 16:45:24.792505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:24.792426 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" path="/var/lib/kubelet/pods/9535c1b3-a8f4-4aca-bded-e847da45c8fa/volumes" Nov 03 16:45:29.063349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:29.063313 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l"] Nov 03 16:45:29.076090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:29.076050 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k"] Nov 03 16:45:29.079890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:29.079865 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13f0e46b-tbm7l"] Nov 03 16:45:29.083281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:29.083258 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4738d5f-2bc8k"] Nov 03 16:45:30.120946 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:30.120903 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq"] Nov 03 16:45:30.140039 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:30.140003 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70bd2474-vrxdq"] Nov 03 16:45:30.792924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:30.792888 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed08ab7-aae5-4ae0-87cd-da29b3024fcd" path="/var/lib/kubelet/pods/1ed08ab7-aae5-4ae0-87cd-da29b3024fcd/volumes" Nov 03 16:45:30.793431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:30.793415 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="513f04c2-e3bf-485c-8597-f30be2b69ddd" path="/var/lib/kubelet/pods/513f04c2-e3bf-485c-8597-f30be2b69ddd/volumes" Nov 03 16:45:30.793951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:30.793936 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a4d895-7d0a-4daa-acfb-7da07325fc73" path="/var/lib/kubelet/pods/99a4d895-7d0a-4daa-acfb-7da07325fc73/volumes" Nov 03 16:45:31.677858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:31.676204 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:45:37.932833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:37.932801 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:45:37.933565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:37.932991 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" containerID="cri-o://cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562" gracePeriod=30 Nov 03 16:45:40.027331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:40.027291 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:45:40.047554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:40.047512 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:45:41.044390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:41.044353 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw"] Nov 03 16:45:41.057073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:41.057039 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37394ffa-m2ttw"] Nov 03 16:45:42.796102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:42.796062 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c6e267-528c-43cc-9a7f-ad0e543c4b4d" path="/var/lib/kubelet/pods/d6c6e267-528c-43cc-9a7f-ad0e543c4b4d/volumes" Nov 03 16:45:43.234274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.234245 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:45:43.442186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442088 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.442342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442186 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.442342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442285 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.442342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442319 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.442509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442350 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.442509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.442398 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.447762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.447719 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.447762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.447761 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.448088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.448034 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\") pod \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\" (UID: \"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54\") " Nov 03 16:45:43.449046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.449016 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:43.449343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.449226 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.449484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.449278 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.449576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.449533 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.450646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.450583 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.454163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.454077 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.454163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.454126 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "kube-api-access-ks75f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:43.457798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.457756 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:43.458023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.457997 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token" pod "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" (UID: "8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:43.550247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550207 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550244 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550255 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550265 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550274 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550281 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550289 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks75f\" (UniqueName: \"kubernetes.io/projected/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-kube-api-access-ks75f\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550297 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.550544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.550309 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:43.944704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.944670 6001 generic.go:356] "Generic (PLEG): container finished" podID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerID="cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562" exitCode=0 Nov 03 16:45:43.945183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.944717 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" Nov 03 16:45:43.945183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.944746 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" event={"ID":"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54","Type":"ContainerDied","Data":"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562"} Nov 03 16:45:43.945183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.944805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr" event={"ID":"8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54","Type":"ContainerDied","Data":"113f12c4532f9a69157cf31b97fd3879b4bde1582fa0833c45ee6b8c83fbb5b3"} Nov 03 16:45:43.945183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.944827 6001 scope.go:122] "RemoveContainer" containerID="cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562" Nov 03 16:45:43.958329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.958300 6001 scope.go:122] "RemoveContainer" containerID="cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562" Nov 03 16:45:43.958686 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:43.958664 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562\": container with ID starting with cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562 not found: ID does not exist" containerID="cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562" Nov 03 16:45:43.958758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:43.958696 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562"} err="failed to get container status \"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562\": rpc error: code = NotFound desc = could not find container \"cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562\": container with ID starting with cfc8cd2e4a17c0e1e3b51c574b71640cf2ce8fa2cf4e6e752c59cb65030f1562 not found: ID does not exist" Nov 03 16:45:44.020290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:44.020252 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:45:44.032712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:44.032674 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-z6krr"] Nov 03 16:45:44.802379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:44.802344 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" path="/var/lib/kubelet/pods/8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54/volumes" Nov 03 16:45:45.028631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.028557 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" podUID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" containerName="payload-processing" containerID="cri-o://6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd" gracePeriod=25 Nov 03 16:45:45.047961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.047929 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" podUID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" containerName="payload-pre-processing" containerID="cri-o://9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194" gracePeriod=25 Nov 03 16:45:45.497594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.497558 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:45:45.541208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.541174 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:45:45.599407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599374 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\") pod \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " Nov 03 16:45:45.599629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599503 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\") pod \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " Nov 03 16:45:45.599708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599659 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\" (UniqueName: \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\") pod \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\" (UID: \"7665b3fc-6789-4027-bdc7-a4c82f9fb085\") " Nov 03 16:45:45.599708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599690 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\" (UniqueName: \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\") pod \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\" (UID: \"5e0c349c-c6ae-4a92-8c92-80ee333ad763\") " Nov 03 16:45:45.599877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599848 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume" pod "5e0c349c-c6ae-4a92-8c92-80ee333ad763" (UID: "5e0c349c-c6ae-4a92-8c92-80ee333ad763"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:45.599941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.599895 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume" pod "7665b3fc-6789-4027-bdc7-a4c82f9fb085" (UID: "7665b3fc-6789-4027-bdc7-a4c82f9fb085"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:45.600434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.600412 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e0c349c-c6ae-4a92-8c92-80ee333ad763-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:45.600434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.600435 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7665b3fc-6789-4027-bdc7-a4c82f9fb085-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:45.601889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.601861 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl" pod "7665b3fc-6789-4027-bdc7-a4c82f9fb085" (UID: "7665b3fc-6789-4027-bdc7-a4c82f9fb085"). InnerVolumeSpecName "kube-api-access-hxcvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:45.602382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.602362 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2" pod "5e0c349c-c6ae-4a92-8c92-80ee333ad763" (UID: "5e0c349c-c6ae-4a92-8c92-80ee333ad763"). InnerVolumeSpecName "kube-api-access-mhpc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:45.702205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.702168 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxcvl\" (UniqueName: \"kubernetes.io/projected/7665b3fc-6789-4027-bdc7-a4c82f9fb085-kube-api-access-hxcvl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:45.702205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.702199 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhpc2\" (UniqueName: \"kubernetes.io/projected/5e0c349c-c6ae-4a92-8c92-80ee333ad763-kube-api-access-mhpc2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:45:45.965541 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.965502 6001 generic.go:356] "Generic (PLEG): container finished" podID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" containerID="9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194" exitCode=0 Nov 03 16:45:45.965740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.965551 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" Nov 03 16:45:45.965740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.965592 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" event={"ID":"5e0c349c-c6ae-4a92-8c92-80ee333ad763","Type":"ContainerDied","Data":"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194"} Nov 03 16:45:45.965740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.965649 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t" event={"ID":"5e0c349c-c6ae-4a92-8c92-80ee333ad763","Type":"ContainerDied","Data":"df131493cd92e942c4e9cdd05e447f0a2e9d09861faaac1abbc5a3216e4414d7"} Nov 03 16:45:45.965740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.965677 6001 scope.go:122] "RemoveContainer" containerID="9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194" Nov 03 16:45:45.967861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.967831 6001 generic.go:356] "Generic (PLEG): container finished" podID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" containerID="6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd" exitCode=0 Nov 03 16:45:45.967973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.967884 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" Nov 03 16:45:45.967973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.967918 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" event={"ID":"7665b3fc-6789-4027-bdc7-a4c82f9fb085","Type":"ContainerDied","Data":"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd"} Nov 03 16:45:45.968060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.967991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9" event={"ID":"7665b3fc-6789-4027-bdc7-a4c82f9fb085","Type":"ContainerDied","Data":"512555d1ff6c2ff52577b3865c994b68bac8383dc3da91be580e4faa298fbaef"} Nov 03 16:45:45.980266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.980235 6001 scope.go:122] "RemoveContainer" containerID="9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194" Nov 03 16:45:45.980706 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:45.980684 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194\": container with ID starting with 9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194 not found: ID does not exist" containerID="9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194" Nov 03 16:45:45.980813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.980717 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194"} err="failed to get container status \"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194\": rpc error: code = NotFound desc = could not find container \"9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194\": container with ID starting with 9c7cbcf86d95b0965b3db3ab1cde541e941ce25149cb8efb11beaf33faa5b194 not found: ID does not exist" Nov 03 16:45:45.980813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.980737 6001 scope.go:122] "RemoveContainer" containerID="6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd" Nov 03 16:45:45.991991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.991971 6001 scope.go:122] "RemoveContainer" containerID="6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd" Nov 03 16:45:45.992302 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:45.992282 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd\": container with ID starting with 6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd not found: ID does not exist" containerID="6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd" Nov 03 16:45:45.992351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:45.992311 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd"} err="failed to get container status \"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd\": rpc error: code = NotFound desc = could not find container \"6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd\": container with ID starting with 6b9507ecafc8f7e50e7ca44e0237cee0dea23010b1aca1170bfbe60c5bb115dd not found: ID does not exist" Nov 03 16:45:46.024729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.024696 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:45:46.028671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.028637 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-73cc79-8558c46477-6gkb9"] Nov 03 16:45:46.036101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.036071 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f"] Nov 03 16:45:46.039690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.039664 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fc0ec4-z6c6f"] Nov 03 16:45:46.062076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.062044 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:45:46.068405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.068378 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-73cc79-6d97c4f667-dqw2t"] Nov 03 16:45:46.341793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.341759 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:45:46.342434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342421 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342435 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342463 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342471 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342492 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" containerName="payload-processing" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342497 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" containerName="payload-processing" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342511 6001 cpu_manager.go:414] "containerMap: removing container" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342517 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e755d4-267b-4203-82b6-67a120f3d3dd" containerName="cleanup" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342533 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342538 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342575 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342580 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342594 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" containerName="payload-pre-processing" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342599 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" containerName="payload-pre-processing" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342704 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24495681-148a-4354-91f1-26089a7b44b8" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342717 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342727 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="709973e0-ca0e-40b6-87a6-af1168e56849" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342733 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342740 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" containerName="payload-processing" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342746 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08c1da5-90ea-43ed-b917-318f687efc81" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342752 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dbe7726-a38e-491f-ab4d-6f2fdbeb9e54" containerName="istio-proxy" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342762 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="87010ed2-0812-491f-ae0a-d39bca083d10" containerName="cleanup" Nov 03 16:45:46.342861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.342769 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" containerName="payload-pre-processing" Nov 03 16:45:46.346007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.345986 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.366683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.366654 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-98e3b9\"" Nov 03 16:45:46.368316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.368287 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-98e3b9-dockercfg-mdj5z\"" Nov 03 16:45:46.371630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.371606 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:45:46.412461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.412415 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btxgq\" (UniqueName: \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.412654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.412578 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.563493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.517605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btxgq\" (UniqueName: \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.563493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.517801 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.563493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.533032 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.563493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.533417 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:45:46.563493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.534869 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9535c1b3-a8f4-4aca-bded-e847da45c8fa" containerName="cleanup" Nov 03 16:45:46.616789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.616689 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.633577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.633546 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btxgq\" (UniqueName: \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\") pod \"payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.643412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.642899 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:45:46.662425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.660626 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:46.743255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.743222 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.743399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.743364 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z7hl\" (UniqueName: \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.794457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.794400 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e0c349c-c6ae-4a92-8c92-80ee333ad763" path="/var/lib/kubelet/pods/5e0c349c-c6ae-4a92-8c92-80ee333ad763/volumes" Nov 03 16:45:46.795158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.795129 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7665b3fc-6789-4027-bdc7-a4c82f9fb085" path="/var/lib/kubelet/pods/7665b3fc-6789-4027-bdc7-a4c82f9fb085/volumes" Nov 03 16:45:46.795777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.795753 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef09499b-da21-4cf2-8d5b-7d4fb1325785" path="/var/lib/kubelet/pods/ef09499b-da21-4cf2-8d5b-7d4fb1325785/volumes" Nov 03 16:45:46.845606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.845567 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.845798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.845688 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z7hl\" (UniqueName: \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.846498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.846475 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.869895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.869832 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z7hl\" (UniqueName: \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\") pod \"payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.965020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.964968 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:46.988140 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:46.988103 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cf6a25_6b42_4bbb_9b02_764df6af86d5.slice/crio-ed608f547f015ac7b65cde236d834feb8eb9c33a4ff82f5734b37f77ad9eb8d4 WatchSource:0}: Error finding container ed608f547f015ac7b65cde236d834feb8eb9c33a4ff82f5734b37f77ad9eb8d4: Status 404 returned error can't find the container with id ed608f547f015ac7b65cde236d834feb8eb9c33a4ff82f5734b37f77ad9eb8d4 Nov 03 16:45:46.988822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:46.988801 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:45:47.281391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.281359 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:45:47.282015 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:47.281988 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba45132_fc0d_41c6_bf4c_c96823d6ff78.slice/crio-4c5175e848a2096c5905fe22971a0070e315ff9644751179d194565a36d01b1f WatchSource:0}: Error finding container 4c5175e848a2096c5905fe22971a0070e315ff9644751179d194565a36d01b1f: Status 404 returned error can't find the container with id 4c5175e848a2096c5905fe22971a0070e315ff9644751179d194565a36d01b1f Nov 03 16:45:47.998741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.997962 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" event={"ID":"5ba45132-fc0d-41c6-bf4c-c96823d6ff78","Type":"ContainerStarted","Data":"f7870f393e5bf0b3255d16f4e5c29f016f50b6fd02fbd2d30c19ae7b54a875ea"} Nov 03 16:45:47.998741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.998309 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:47.998741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.998348 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" event={"ID":"5ba45132-fc0d-41c6-bf4c-c96823d6ff78","Type":"ContainerStarted","Data":"4c5175e848a2096c5905fe22971a0070e315ff9644751179d194565a36d01b1f"} Nov 03 16:45:47.999907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.999878 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" event={"ID":"64cf6a25-6b42-4bbb-9b02-764df6af86d5","Type":"ContainerStarted","Data":"2c5566085be9cb9a87739374a7220469ff2cc2a35dcfd41120754de43f1ce9d7"} Nov 03 16:45:48.000051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.999924 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" event={"ID":"64cf6a25-6b42-4bbb-9b02-764df6af86d5","Type":"ContainerStarted","Data":"ed608f547f015ac7b65cde236d834feb8eb9c33a4ff82f5734b37f77ad9eb8d4"} Nov 03 16:45:48.000051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:47.999979 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:48.046782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:48.046719 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" podStartSLOduration=2.046699362 podStartE2EDuration="2.046699362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:48.041534002 +0000 UTC m=+1843.374747573" watchObservedRunningTime="2026-11-03 16:45:48.046699362 +0000 UTC m=+1843.379912933" Nov 03 16:45:48.088189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:48.088135 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" podStartSLOduration=2.08812217 podStartE2EDuration="2.08812217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:48.085113207 +0000 UTC m=+1843.418326777" watchObservedRunningTime="2026-11-03 16:45:48.08812217 +0000 UTC m=+1843.421335738" Nov 03 16:45:52.254231 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:45:52.254182 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:45:57.708096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.708058 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:45:57.711476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.711431 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.733063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.733030 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:45:57.741714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.741684 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-624649-dockercfg-shwqn\"" Nov 03 16:45:57.743026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.742991 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-624649\"" Nov 03 16:45:57.781532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.781090 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:45:57.786597 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.786562 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.813852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.813768 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:45:57.865075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.865035 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.865363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.865342 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt8l5\" (UniqueName: \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.865480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.865403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txh9h\" (UniqueName: \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.865598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.865567 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.967802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.967699 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txh9h\" (UniqueName: \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.967802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.967761 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.968042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.967948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.968172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.968156 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt8l5\" (UniqueName: \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.968795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.968768 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:57.968925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.968814 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.992656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.992618 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txh9h\" (UniqueName: \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\") pod \"payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:57.992853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:57.992752 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt8l5\" (UniqueName: \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\") pod \"payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:58.025517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:58.025479 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:58.133032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:58.132719 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:58.351288 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:58.351242 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad3498f_715c_4be0_83a2_33800ea65757.slice/crio-2db6c50eb14f4266b688b09ba1ddb240fbdee83a2e23648cdf44a9c54aba8aee WatchSource:0}: Error finding container 2db6c50eb14f4266b688b09ba1ddb240fbdee83a2e23648cdf44a9c54aba8aee: Status 404 returned error can't find the container with id 2db6c50eb14f4266b688b09ba1ddb240fbdee83a2e23648cdf44a9c54aba8aee Nov 03 16:45:58.353558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:58.353525 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:45:58.512245 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:45:58.512207 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45c94a8_b00e_4509_8c42_5d2eb3b50346.slice/crio-ef1db34d1a9d2dff6cd535a098e8218e010ac15d9e6184642c11a2b79412fcd9 WatchSource:0}: Error finding container ef1db34d1a9d2dff6cd535a098e8218e010ac15d9e6184642c11a2b79412fcd9: Status 404 returned error can't find the container with id ef1db34d1a9d2dff6cd535a098e8218e010ac15d9e6184642c11a2b79412fcd9 Nov 03 16:45:58.513565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:58.513539 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:45:59.012943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.012914 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:45:59.013379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.012983 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:45:59.178069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.178030 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" event={"ID":"4ad3498f-715c-4be0-83a2-33800ea65757","Type":"ContainerStarted","Data":"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347"} Nov 03 16:45:59.178335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.178091 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" event={"ID":"4ad3498f-715c-4be0-83a2-33800ea65757","Type":"ContainerStarted","Data":"2db6c50eb14f4266b688b09ba1ddb240fbdee83a2e23648cdf44a9c54aba8aee"} Nov 03 16:45:59.178335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.178118 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:45:59.180046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.180015 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" event={"ID":"d45c94a8-b00e-4509-8c42-5d2eb3b50346","Type":"ContainerStarted","Data":"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825"} Nov 03 16:45:59.180191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.180057 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" event={"ID":"d45c94a8-b00e-4509-8c42-5d2eb3b50346","Type":"ContainerStarted","Data":"ef1db34d1a9d2dff6cd535a098e8218e010ac15d9e6184642c11a2b79412fcd9"} Nov 03 16:45:59.180191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.180111 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:45:59.230729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.230664 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" podStartSLOduration=2.230644629 podStartE2EDuration="2.230644629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:59.227005785 +0000 UTC m=+1854.560219355" watchObservedRunningTime="2026-11-03 16:45:59.230644629 +0000 UTC m=+1854.563858198" Nov 03 16:45:59.276060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:45:59.275967 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" podStartSLOduration=2.275952074 podStartE2EDuration="2.275952074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:59.272270123 +0000 UTC m=+1854.605483693" watchObservedRunningTime="2026-11-03 16:45:59.275952074 +0000 UTC m=+1854.609165644" Nov 03 16:46:02.211419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:02.211381 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:46:02.244193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:02.244150 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:46:07.211888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.211838 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" podUID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" containerName="payload-processing" containerID="cri-o://0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce" gracePeriod=25 Nov 03 16:46:07.245083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.245024 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" podUID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" containerName="payload-pre-processing" containerID="cri-o://6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed" gracePeriod=25 Nov 03 16:46:07.573346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.573318 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:46:07.577558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.577529 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.668788 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\") pod \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.668866 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\" (UniqueName: \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\") pod \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\" (UID: \"38ddb60f-6ac9-4282-ad23-5b99bbcea118\") " Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.669018 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\" (UniqueName: \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\") pod \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.669532 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\") pod \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\" (UID: \"634e4f1d-2213-4eae-b7c1-8ac92671cd59\") " Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.670794 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume" pod "634e4f1d-2213-4eae-b7c1-8ac92671cd59" (UID: "634e4f1d-2213-4eae-b7c1-8ac92671cd59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.671517 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume" pod "38ddb60f-6ac9-4282-ad23-5b99bbcea118" (UID: "38ddb60f-6ac9-4282-ad23-5b99bbcea118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.674503 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr" pod "634e4f1d-2213-4eae-b7c1-8ac92671cd59" (UID: "634e4f1d-2213-4eae-b7c1-8ac92671cd59"). InnerVolumeSpecName "kube-api-access-tbwmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:07.687466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.674552 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44" pod "38ddb60f-6ac9-4282-ad23-5b99bbcea118" (UID: "38ddb60f-6ac9-4282-ad23-5b99bbcea118"). InnerVolumeSpecName "kube-api-access-gpj44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:07.771773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.771660 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634e4f1d-2213-4eae-b7c1-8ac92671cd59-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:07.771773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.771708 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38ddb60f-6ac9-4282-ad23-5b99bbcea118-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:07.771773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.771722 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpj44\" (UniqueName: \"kubernetes.io/projected/38ddb60f-6ac9-4282-ad23-5b99bbcea118-kube-api-access-gpj44\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:07.771773 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:07.771735 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwmr\" (UniqueName: \"kubernetes.io/projected/634e4f1d-2213-4eae-b7c1-8ac92671cd59-kube-api-access-tbwmr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:08.262297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.262258 6001 generic.go:356] "Generic (PLEG): container finished" podID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" containerID="6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed" exitCode=0 Nov 03 16:46:08.262855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.262310 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" Nov 03 16:46:08.262855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.262323 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" event={"ID":"38ddb60f-6ac9-4282-ad23-5b99bbcea118","Type":"ContainerDied","Data":"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed"} Nov 03 16:46:08.262855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.262381 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27" event={"ID":"38ddb60f-6ac9-4282-ad23-5b99bbcea118","Type":"ContainerDied","Data":"0b036f0ac60cd1c1c0f805c00e797caa79095e9e71f2714056679a905f868bff"} Nov 03 16:46:08.262855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.262403 6001 scope.go:122] "RemoveContainer" containerID="6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed" Nov 03 16:46:08.264433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.264407 6001 generic.go:356] "Generic (PLEG): container finished" podID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" containerID="0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce" exitCode=0 Nov 03 16:46:08.264703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.264588 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" event={"ID":"634e4f1d-2213-4eae-b7c1-8ac92671cd59","Type":"ContainerDied","Data":"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce"} Nov 03 16:46:08.264703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.264598 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" Nov 03 16:46:08.264703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.264642 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z" event={"ID":"634e4f1d-2213-4eae-b7c1-8ac92671cd59","Type":"ContainerDied","Data":"d157a122fbb3d670a7cc79de37f06bad52a60ae1db4037601bcd7920047b97be"} Nov 03 16:46:08.282408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.282378 6001 scope.go:122] "RemoveContainer" containerID="6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed" Nov 03 16:46:08.284771 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:08.284720 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed\": container with ID starting with 6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed not found: ID does not exist" containerID="6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed" Nov 03 16:46:08.284948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.284770 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed"} err="failed to get container status \"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed\": rpc error: code = NotFound desc = could not find container \"6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed\": container with ID starting with 6f8ac02e4a00307cc0f93d845b65c1f385c286e5e359854cdf18e5df5e28bfed not found: ID does not exist" Nov 03 16:46:08.284948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.284792 6001 scope.go:122] "RemoveContainer" containerID="0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce" Nov 03 16:46:08.308547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.308517 6001 scope.go:122] "RemoveContainer" containerID="0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce" Nov 03 16:46:08.309162 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:08.309127 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce\": container with ID starting with 0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce not found: ID does not exist" containerID="0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce" Nov 03 16:46:08.309282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.309169 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce"} err="failed to get container status \"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce\": rpc error: code = NotFound desc = could not find container \"0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce\": container with ID starting with 0f5e0807e388c78af304173d1ac796a5f380e5e534f006ba2297bf263c5cc9ce not found: ID does not exist" Nov 03 16:46:08.338573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.338502 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:46:08.347031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.346989 6001 scope.go:122] "RemoveContainer" containerID="27c553cb34205e72968d123122b118fb7e32727246ed28d76a9444e874280606" Nov 03 16:46:08.351205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.351176 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-db3c6b-59657644b7-l8l27"] Nov 03 16:46:08.359565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.359542 6001 scope.go:122] "RemoveContainer" containerID="449eb97a377deb4dc5208fbc300b81696a15c06b45a5eb0af2f94cfdb32e0d76" Nov 03 16:46:08.372340 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.372279 6001 scope.go:122] "RemoveContainer" containerID="d7fb0ab5c9a81e4c7dc9e283bf60e1697692f723ae40c49872c3e75d10bf7c2b" Nov 03 16:46:08.378613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.378580 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:46:08.384406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.384372 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-db3c6b-6b57d8899-fhs6z"] Nov 03 16:46:08.388699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.388672 6001 scope.go:122] "RemoveContainer" containerID="5de119ec9f3b15544bb5d9cf255ff24fa75fb1a4a9bf197c7b6e37bee9115161" Nov 03 16:46:08.401502 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.401430 6001 scope.go:122] "RemoveContainer" containerID="09a727110d4e679817b700f7a2b226881ca5fb28f14024f78422ff028cd54aeb" Nov 03 16:46:08.414584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.414539 6001 scope.go:122] "RemoveContainer" containerID="1e4578f6194803bc18f25daa5adf6a0b11d4697f674ae96209d8ff5fe7a60811" Nov 03 16:46:08.427418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.427392 6001 scope.go:122] "RemoveContainer" containerID="d76c88d84eea130650e2efd51d23d1cfcbaf80ee3b07d33b679fef594dad6de3" Nov 03 16:46:08.441881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.441851 6001 scope.go:122] "RemoveContainer" containerID="ee974a0b360e91b6b44a0d45fd8475c61d2873bc0a988f4a8a57d71abc3b33e1" Nov 03 16:46:08.455473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.455423 6001 scope.go:122] "RemoveContainer" containerID="09cc5a7749829e0b17a5991bd56bc226737b5591c513939493be2de8258a91a0" Nov 03 16:46:08.466188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.466164 6001 scope.go:122] "RemoveContainer" containerID="d3ff1073d0aaef100b191bb6a4e5369dbb611c87c5d72bbda65d741d99e26032" Nov 03 16:46:08.794096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.794009 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" path="/var/lib/kubelet/pods/38ddb60f-6ac9-4282-ad23-5b99bbcea118/volumes" Nov 03 16:46:08.794519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:08.794504 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" path="/var/lib/kubelet/pods/634e4f1d-2213-4eae-b7c1-8ac92671cd59/volumes" Nov 03 16:46:10.191634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:10.191598 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:46:10.192322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:10.191694 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:46:13.489464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:13.485414 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:46:17.740147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.740114 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:46:17.740618 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.740327 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" containerID="cri-o://f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0" gracePeriod=30 Nov 03 16:46:17.753303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.753239 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:46:17.753760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.753722 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" containerID="cri-o://afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033" gracePeriod=30 Nov 03 16:46:17.757260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.757232 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:46:17.757577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:17.757546 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" containerID="cri-o://42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771" gracePeriod=30 Nov 03 16:46:18.261505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.261468 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:46:18.298679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.298641 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:46:18.486687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.486648 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" podUID="08df94b8-dca8-4f34-9b09-7f2d14804142" containerName="payload-processing" containerID="cri-o://d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d" gracePeriod=25 Nov 03 16:46:18.914655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.914623 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:46:18.945924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.945884 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\" (UniqueName: \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\") pod \"08df94b8-dca8-4f34-9b09-7f2d14804142\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " Nov 03 16:46:18.946093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.945992 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\") pod \"08df94b8-dca8-4f34-9b09-7f2d14804142\" (UID: \"08df94b8-dca8-4f34-9b09-7f2d14804142\") " Nov 03 16:46:18.946358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.946334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume" pod "08df94b8-dca8-4f34-9b09-7f2d14804142" (UID: "08df94b8-dca8-4f34-9b09-7f2d14804142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:18.946674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.946651 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08df94b8-dca8-4f34-9b09-7f2d14804142-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:18.948222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:18.948201 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz" pod "08df94b8-dca8-4f34-9b09-7f2d14804142" (UID: "08df94b8-dca8-4f34-9b09-7f2d14804142"). InnerVolumeSpecName "kube-api-access-2ldnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:19.048759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.048684 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ldnz\" (UniqueName: \"kubernetes.io/projected/08df94b8-dca8-4f34-9b09-7f2d14804142-kube-api-access-2ldnz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:19.373227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.373196 6001 generic.go:356] "Generic (PLEG): container finished" podID="08df94b8-dca8-4f34-9b09-7f2d14804142" containerID="d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d" exitCode=0 Nov 03 16:46:19.373420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.373244 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" Nov 03 16:46:19.373420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.373274 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" event={"ID":"08df94b8-dca8-4f34-9b09-7f2d14804142","Type":"ContainerDied","Data":"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d"} Nov 03 16:46:19.373420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.373336 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn" event={"ID":"08df94b8-dca8-4f34-9b09-7f2d14804142","Type":"ContainerDied","Data":"b36fe22fc8484e6ec6bde5d0e731a26b05c96ea7a7c40059d17ef80e228adee3"} Nov 03 16:46:19.373420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.373366 6001 scope.go:122] "RemoveContainer" containerID="d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d" Nov 03 16:46:19.386304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.386277 6001 scope.go:122] "RemoveContainer" containerID="d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d" Nov 03 16:46:19.386773 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:19.386749 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d\": container with ID starting with d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d not found: ID does not exist" containerID="d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d" Nov 03 16:46:19.386888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.386781 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d"} err="failed to get container status \"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d\": rpc error: code = NotFound desc = could not find container \"d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d\": container with ID starting with d34d188863a3e5a15e84cbc1c3fe67a48d2e1646846abcb200cadda72370be6d not found: ID does not exist" Nov 03 16:46:19.454025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.453930 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:46:19.492722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:19.492681 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-8f4b49d49-gzwjn"] Nov 03 16:46:20.792291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:20.792246 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08df94b8-dca8-4f34-9b09-7f2d14804142" path="/var/lib/kubelet/pods/08df94b8-dca8-4f34-9b09-7f2d14804142/volumes" Nov 03 16:46:22.262547 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:22.262507 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:46:23.063306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.063273 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:46:23.117435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117397 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.117633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117513 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.117633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117565 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.117633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117616 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.117823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117682 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.117888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.117855 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.118103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.118084 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.118195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.118128 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.118195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.118188 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\") pod \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\" (UID: \"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3\") " Nov 03 16:46:23.119231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.119200 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.119753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.119721 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.119933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.119912 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.120634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.120596 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.120861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.120834 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:23.122072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.122041 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.122831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.122795 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "kube-api-access-kw6fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.146907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.146843 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.195068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.177237 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo" pod "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" (UID: "5d8d5dda-fb6f-4913-9f87-8410d9ac54b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:23.202615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.202582 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:46:23.208974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.206759 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:46:23.220377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220343 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220375 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw6fh\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-kube-api-access-kw6fh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220390 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220402 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220412 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220420 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220428 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220436 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.220707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.220498 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.262070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.262024 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" podUID="5685af88-fea8-45af-892d-5c08b750edc0" containerName="payload-processing" containerID="cri-o://9009104a84bbd349fffd2ea6ea3b7aa7e956749b8caa7fa1565bc664d8dc661f" gracePeriod=25 Nov 03 16:46:23.294678 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:23.294564 6001 logging.go:55] [core] [Channel #2486 SubChannel #2488]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.136:50052", ServerName: "10.134.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.136:50052: connect: connection refused" Nov 03 16:46:23.300220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.300181 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" podUID="e12a7cae-1690-4e5c-90aa-5a754e51302d" containerName="payload-pre-processing" containerID="cri-o://cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea" gracePeriod=25 Nov 03 16:46:23.324964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.324921 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325047 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325080 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325111 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325161 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325188 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325223 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325261 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325285 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325313 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325412 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325567 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325592 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325625 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.325785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325653 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.325785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325719 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\") pod \"8cb149bb-32f8-438c-842b-364c222becfc\" (UID: \"8cb149bb-32f8-438c-842b-364c222becfc\") " Nov 03 16:46:23.326107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.325892 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\") pod \"8b713fe4-991b-4e13-be87-4bdab0c74e69\" (UID: \"8b713fe4-991b-4e13-be87-4bdab0c74e69\") " Nov 03 16:46:23.328671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.328432 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.329078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.328905 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.329078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.329059 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.329326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.328607 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:23.329462 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.328702 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.329587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.328766 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:23.329665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.329427 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.329793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.329774 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.330199 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.330171 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.330390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.330370 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.333273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.333206 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:23.333273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.333227 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.333476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.333362 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "kube-api-access-2bmcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.334891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.334866 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.335155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.335120 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.335240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.335220 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:23.335633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.335608 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8" pod "8cb149bb-32f8-438c-842b-364c222becfc" (UID: "8cb149bb-32f8-438c-842b-364c222becfc"). InnerVolumeSpecName "kube-api-access-k8ds8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.335712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.335652 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy" pod "8b713fe4-991b-4e13-be87-4bdab0c74e69" (UID: "8b713fe4-991b-4e13-be87-4bdab0c74e69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:23.407273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.407200 6001 generic.go:356] "Generic (PLEG): container finished" podID="5685af88-fea8-45af-892d-5c08b750edc0" containerID="9009104a84bbd349fffd2ea6ea3b7aa7e956749b8caa7fa1565bc664d8dc661f" exitCode=0 Nov 03 16:46:23.407348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.407277 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" event={"ID":"5685af88-fea8-45af-892d-5c08b750edc0","Type":"ContainerDied","Data":"9009104a84bbd349fffd2ea6ea3b7aa7e956749b8caa7fa1565bc664d8dc661f"} Nov 03 16:46:23.409413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.409373 6001 generic.go:356] "Generic (PLEG): container finished" podID="8cb149bb-32f8-438c-842b-364c222becfc" containerID="afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033" exitCode=0 Nov 03 16:46:23.409554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.409500 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" event={"ID":"8cb149bb-32f8-438c-842b-364c222becfc","Type":"ContainerDied","Data":"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033"} Nov 03 16:46:23.409554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.409532 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" Nov 03 16:46:23.409681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.409563 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" event={"ID":"8cb149bb-32f8-438c-842b-364c222becfc","Type":"ContainerDied","Data":"11e90a9fe2a22f858ef7b740e47f48b4728c8c32413711347ae0a9ca7130912a"} Nov 03 16:46:23.409681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.409598 6001 scope.go:122] "RemoveContainer" containerID="afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033" Nov 03 16:46:23.411274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.411249 6001 generic.go:356] "Generic (PLEG): container finished" podID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerID="42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771" exitCode=0 Nov 03 16:46:23.411392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.411341 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" event={"ID":"8b713fe4-991b-4e13-be87-4bdab0c74e69","Type":"ContainerDied","Data":"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771"} Nov 03 16:46:23.411392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.411382 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" event={"ID":"8b713fe4-991b-4e13-be87-4bdab0c74e69","Type":"ContainerDied","Data":"2ae9dce01dc364b5f5d39ec3b85cedf3e2cf92167b4adbd8579d36614e0ab206"} Nov 03 16:46:23.411590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.411467 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t" Nov 03 16:46:23.418696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.418661 6001 generic.go:356] "Generic (PLEG): container finished" podID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerID="f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0" exitCode=0 Nov 03 16:46:23.419202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.418720 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" event={"ID":"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3","Type":"ContainerDied","Data":"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0"} Nov 03 16:46:23.419202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.418769 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" event={"ID":"5d8d5dda-fb6f-4913-9f87-8410d9ac54b3","Type":"ContainerDied","Data":"c1601b602fc758236ce74dd20892d42a158fa355e509964fda5764b95e87a00e"} Nov 03 16:46:23.419202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.418793 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5" Nov 03 16:46:23.428523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428436 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428516 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ds8\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-kube-api-access-k8ds8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428532 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428543 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428552 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428560 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cb149bb-32f8-438c-842b-364c222becfc-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428567 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428575 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428583 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428592 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428600 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428609 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428617 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cb149bb-32f8-438c-842b-364c222becfc-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428625 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b713fe4-991b-4e13-be87-4bdab0c74e69-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428633 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cb149bb-32f8-438c-842b-364c222becfc-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428640 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b713fe4-991b-4e13-be87-4bdab0c74e69-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428648 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cb149bb-32f8-438c-842b-364c222becfc-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.428713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.428656 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmcv\" (UniqueName: \"kubernetes.io/projected/8b713fe4-991b-4e13-be87-4bdab0c74e69-kube-api-access-2bmcv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.431276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.431033 6001 scope.go:122] "RemoveContainer" containerID="afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033" Nov 03 16:46:23.431836 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:23.431810 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033\": container with ID starting with afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033 not found: ID does not exist" containerID="afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033" Nov 03 16:46:23.431934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.431848 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033"} err="failed to get container status \"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033\": rpc error: code = NotFound desc = could not find container \"afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033\": container with ID starting with afa05ba1bea8fe70314590338ed63db9c4b8afe827a9d3f97865684bec8b6033 not found: ID does not exist" Nov 03 16:46:23.431934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.431874 6001 scope.go:122] "RemoveContainer" containerID="42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771" Nov 03 16:46:23.454941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.454911 6001 scope.go:122] "RemoveContainer" containerID="42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771" Nov 03 16:46:23.455653 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:23.455623 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771\": container with ID starting with 42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771 not found: ID does not exist" containerID="42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771" Nov 03 16:46:23.455784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.455664 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771"} err="failed to get container status \"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771\": rpc error: code = NotFound desc = could not find container \"42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771\": container with ID starting with 42a0e494839043dc3902c6abdbffe101ee99a04408c52944c192a241289af771 not found: ID does not exist" Nov 03 16:46:23.455784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.455691 6001 scope.go:122] "RemoveContainer" containerID="f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0" Nov 03 16:46:23.468978 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.468951 6001 scope.go:122] "RemoveContainer" containerID="f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0" Nov 03 16:46:23.469312 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:23.469287 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0\": container with ID starting with f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0 not found: ID does not exist" containerID="f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0" Nov 03 16:46:23.469359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.469322 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0"} err="failed to get container status \"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0\": rpc error: code = NotFound desc = could not find container \"f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0\": container with ID starting with f8768b8ee60db4ed91a96bd65daab49c5bce9e982a847b1eda6de9531bc35dc0 not found: ID does not exist" Nov 03 16:46:23.489125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.489096 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:46:23.494013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.493975 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-gwp5t"] Nov 03 16:46:23.604289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.598929 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:46:23.607163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.607129 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb"] Nov 03 16:46:23.636572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.636439 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:46:23.640469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.640419 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:46:23.645049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.645005 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-9cdx5"] Nov 03 16:46:23.716243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.716211 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:46:23.750577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.750542 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\" (UniqueName: \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\") pod \"5685af88-fea8-45af-892d-5c08b750edc0\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " Nov 03 16:46:23.750777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.750705 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\") pod \"5685af88-fea8-45af-892d-5c08b750edc0\" (UID: \"5685af88-fea8-45af-892d-5c08b750edc0\") " Nov 03 16:46:23.751096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.751069 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume" pod "5685af88-fea8-45af-892d-5c08b750edc0" (UID: "5685af88-fea8-45af-892d-5c08b750edc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:23.751205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.751192 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5685af88-fea8-45af-892d-5c08b750edc0-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.752707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.752687 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4" pod "5685af88-fea8-45af-892d-5c08b750edc0" (UID: "5685af88-fea8-45af-892d-5c08b750edc0"). InnerVolumeSpecName "kube-api-access-4krx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.852717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.852672 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\") pod \"e12a7cae-1690-4e5c-90aa-5a754e51302d\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " Nov 03 16:46:23.852887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.852814 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\") pod \"e12a7cae-1690-4e5c-90aa-5a754e51302d\" (UID: \"e12a7cae-1690-4e5c-90aa-5a754e51302d\") " Nov 03 16:46:23.853130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.853101 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume" pod "e12a7cae-1690-4e5c-90aa-5a754e51302d" (UID: "e12a7cae-1690-4e5c-90aa-5a754e51302d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:23.853293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.853278 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4krx4\" (UniqueName: \"kubernetes.io/projected/5685af88-fea8-45af-892d-5c08b750edc0-kube-api-access-4krx4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.853330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.853296 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e12a7cae-1690-4e5c-90aa-5a754e51302d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:23.855072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.855028 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs" pod "e12a7cae-1690-4e5c-90aa-5a754e51302d" (UID: "e12a7cae-1690-4e5c-90aa-5a754e51302d"). InnerVolumeSpecName "kube-api-access-l68bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:23.954321 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:23.954274 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68bs\" (UniqueName: \"kubernetes.io/projected/e12a7cae-1690-4e5c-90aa-5a754e51302d-kube-api-access-l68bs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:24.165271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.165148 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:46:24.165271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.165214 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-hnmjb" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:46:24.294699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.294654 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" podUID="5685af88-fea8-45af-892d-5c08b750edc0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.136:50052\" within 1s: context deadline exceeded" Nov 03 16:46:24.444838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.444755 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" Nov 03 16:46:24.444838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.444782 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8" event={"ID":"5685af88-fea8-45af-892d-5c08b750edc0","Type":"ContainerDied","Data":"843fa5a54fe44f68988d40659675807f490d85623fec21dbb82e7845a0b7414f"} Nov 03 16:46:24.445061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.444891 6001 scope.go:122] "RemoveContainer" containerID="9009104a84bbd349fffd2ea6ea3b7aa7e956749b8caa7fa1565bc664d8dc661f" Nov 03 16:46:24.448437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.448407 6001 generic.go:356] "Generic (PLEG): container finished" podID="e12a7cae-1690-4e5c-90aa-5a754e51302d" containerID="cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea" exitCode=0 Nov 03 16:46:24.448954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.448626 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" event={"ID":"e12a7cae-1690-4e5c-90aa-5a754e51302d","Type":"ContainerDied","Data":"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea"} Nov 03 16:46:24.448954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.448678 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" event={"ID":"e12a7cae-1690-4e5c-90aa-5a754e51302d","Type":"ContainerDied","Data":"2f797756a6596dd2af755e15940f0e10431e2c44e834062327f9f978d172d893"} Nov 03 16:46:24.449372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.449356 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw" Nov 03 16:46:24.462818 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.462795 6001 scope.go:122] "RemoveContainer" containerID="cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea" Nov 03 16:46:24.473158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.473139 6001 scope.go:122] "RemoveContainer" containerID="cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea" Nov 03 16:46:24.473452 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:24.473424 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea\": container with ID starting with cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea not found: ID does not exist" containerID="cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea" Nov 03 16:46:24.473509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.473483 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea"} err="failed to get container status \"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea\": rpc error: code = NotFound desc = could not find container \"cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea\": container with ID starting with cef5c8edd0860dacb8f51c1f6ecc4efafc5699471653fa83d753d448959fe4ea not found: ID does not exist" Nov 03 16:46:24.518225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.518192 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:46:24.536154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.536123 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2aa2ad-7cc8bc7f95-fh9n8"] Nov 03 16:46:24.571625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.571583 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:46:24.577799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.577763 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2aa2ad-66d4c466cf-xqzmw"] Nov 03 16:46:24.792662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.792629 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5685af88-fea8-45af-892d-5c08b750edc0" path="/var/lib/kubelet/pods/5685af88-fea8-45af-892d-5c08b750edc0/volumes" Nov 03 16:46:24.793221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.793205 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" path="/var/lib/kubelet/pods/5d8d5dda-fb6f-4913-9f87-8410d9ac54b3/volumes" Nov 03 16:46:24.793739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.793724 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" path="/var/lib/kubelet/pods/8b713fe4-991b-4e13-be87-4bdab0c74e69/volumes" Nov 03 16:46:24.794577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.794562 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb149bb-32f8-438c-842b-364c222becfc" path="/var/lib/kubelet/pods/8cb149bb-32f8-438c-842b-364c222becfc/volumes" Nov 03 16:46:24.794982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:24.794968 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12a7cae-1690-4e5c-90aa-5a754e51302d" path="/var/lib/kubelet/pods/e12a7cae-1690-4e5c-90aa-5a754e51302d/volumes" Nov 03 16:46:27.877401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.877354 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:46:27.878195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878167 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878196 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878219 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e12a7cae-1690-4e5c-90aa-5a754e51302d" containerName="payload-pre-processing" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878226 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12a7cae-1690-4e5c-90aa-5a754e51302d" containerName="payload-pre-processing" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878235 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878241 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" Nov 03 16:46:27.878260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878257 6001 cpu_manager.go:414] "containerMap: removing container" podUID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" containerName="payload-pre-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878265 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" containerName="payload-pre-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878273 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5685af88-fea8-45af-892d-5c08b750edc0" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878277 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5685af88-fea8-45af-892d-5c08b750edc0" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878293 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878298 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878311 6001 cpu_manager.go:414] "containerMap: removing container" podUID="08df94b8-dca8-4f34-9b09-7f2d14804142" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878315 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08df94b8-dca8-4f34-9b09-7f2d14804142" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878324 6001 cpu_manager.go:414] "containerMap: removing container" podUID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878329 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878436 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="38ddb60f-6ac9-4282-ad23-5b99bbcea118" containerName="payload-pre-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878467 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5685af88-fea8-45af-892d-5c08b750edc0" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878484 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb149bb-32f8-438c-842b-364c222becfc" containerName="istio-proxy" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878497 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="08df94b8-dca8-4f34-9b09-7f2d14804142" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878511 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="634e4f1d-2213-4eae-b7c1-8ac92671cd59" containerName="payload-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878523 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8d5dda-fb6f-4913-9f87-8410d9ac54b3" containerName="istio-proxy" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878530 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12a7cae-1690-4e5c-90aa-5a754e51302d" containerName="payload-pre-processing" Nov 03 16:46:27.879057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.878536 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b713fe4-991b-4e13-be87-4bdab0c74e69" containerName="istio-proxy" Nov 03 16:46:27.881359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.881333 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:27.901710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.901681 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0a112-dockercfg-x5pq8\"" Nov 03 16:46:27.901872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.901716 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0a112\"" Nov 03 16:46:27.920681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:27.920646 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:46:28.031627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.031288 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:46:28.057470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.052472 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcm4k\" (UniqueName: \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.057470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.052820 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.074468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.069790 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:46:28.074468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.069984 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.153956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.153850 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.154153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.153957 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcm4k\" (UniqueName: \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.154153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.154065 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xst\" (UniqueName: \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.154248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.154159 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.154918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.154893 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.179168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.179126 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcm4k\" (UniqueName: \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.196661 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.196624 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:28.256236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.256193 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xst\" (UniqueName: \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.256493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.256474 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.257659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.257629 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.287521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.287485 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xst\" (UniqueName: \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\") pod \"payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.423850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.423748 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:28.514704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.514675 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:46:28.728954 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:28.728921 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2655094a_621d_4a61_986d_257e16715040.slice/crio-77f2371ae6f3962dc815e138150f4469e6547da56acf3420f8b3d0bd62142d51 WatchSource:0}: Error finding container 77f2371ae6f3962dc815e138150f4469e6547da56acf3420f8b3d0bd62142d51: Status 404 returned error can't find the container with id 77f2371ae6f3962dc815e138150f4469e6547da56acf3420f8b3d0bd62142d51 Nov 03 16:46:28.729117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:28.729051 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:46:29.496669 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.496624 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" event={"ID":"2655094a-621d-4a61-986d-257e16715040","Type":"ContainerStarted","Data":"53c946c40437088e268e6c5f97a51ffc268528234da57bc710f04da661b344b0"} Nov 03 16:46:29.497241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.496685 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" event={"ID":"2655094a-621d-4a61-986d-257e16715040","Type":"ContainerStarted","Data":"77f2371ae6f3962dc815e138150f4469e6547da56acf3420f8b3d0bd62142d51"} Nov 03 16:46:29.497241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.496814 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:29.498656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.498627 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" event={"ID":"c54964d5-88b6-498e-b9e2-8d0b92cd88e7","Type":"ContainerStarted","Data":"cc3e8641ec3112a9e3b3829cc632d9e8b8ada436e5ab58d41515161ebd46e442"} Nov 03 16:46:29.498796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.498670 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" event={"ID":"c54964d5-88b6-498e-b9e2-8d0b92cd88e7","Type":"ContainerStarted","Data":"0fcd03d9dbb81b19359a83c152c4f064965f211048795e3a0f41f1c494ecc01e"} Nov 03 16:46:29.498796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.498714 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:29.549798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.549734 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" podStartSLOduration=2.5497198 podStartE2EDuration="2.5497198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:29.546856614 +0000 UTC m=+1884.880070185" watchObservedRunningTime="2026-11-03 16:46:29.5497198 +0000 UTC m=+1884.882933366" Nov 03 16:46:29.596217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:29.596151 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" podStartSLOduration=2.596132021 podStartE2EDuration="2.596132021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:29.592430825 +0000 UTC m=+1884.925644394" watchObservedRunningTime="2026-11-03 16:46:29.596132021 +0000 UTC m=+1884.929345591" Nov 03 16:46:31.129831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.129790 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:46:31.130326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.129998 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" containerID="cri-o://62fe059155c73051583e03148466b6d7c42acce84957bc433d8adf24e7e55e48" gracePeriod=30 Nov 03 16:46:31.142979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.142944 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:46:31.143172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.143151 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" containerID="cri-o://cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8" gracePeriod=30 Nov 03 16:46:31.156509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.156426 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:46:31.157007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:31.156766 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" containerID="cri-o://557eaf993f5cdf0609790ce0ebc1faf12fcc8e97f72ee67e948b771cd5920031" gracePeriod=30 Nov 03 16:46:34.035235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.035195 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:46:34.039557 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.039537 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.059795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.059747 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:46:34.068761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.068727 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-53bd50-dockercfg-8v947\"" Nov 03 16:46:34.069015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.068925 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-53bd50\"" Nov 03 16:46:34.151183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.151144 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.151528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.151504 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wr2\" (UniqueName: \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.181479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.181419 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:46:34.185139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.185117 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.206563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.206527 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:46:34.253268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.253227 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.253482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.253280 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wr2\" (UniqueName: \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.253482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.253410 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.253612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.253500 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxnr\" (UniqueName: \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.254267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.254241 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.278634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.278597 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wr2\" (UniqueName: \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.355112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.355010 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.355112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.355079 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxnr\" (UniqueName: \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.355786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.355765 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.356571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.356549 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:34.380480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.379497 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxnr\" (UniqueName: \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\") pod \"payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.499019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.498985 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:34.669352 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:34.669310 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a6628a_7e6e_46c3_b076_b98fb9dfdefe.slice/crio-471546825d390c71e0e2fd956bed7101c9a76f5c29a6461538dfe7373cc44cdb WatchSource:0}: Error finding container 471546825d390c71e0e2fd956bed7101c9a76f5c29a6461538dfe7373cc44cdb: Status 404 returned error can't find the container with id 471546825d390c71e0e2fd956bed7101c9a76f5c29a6461538dfe7373cc44cdb Nov 03 16:46:34.669514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.669352 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:46:34.812032 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:34.812003 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c842719_571c_4cb7_a442_a88ad7ee53c7.slice/crio-10828ad6c00bd8c252c80a375c6d02baf1a67becd5a1c64d3355bac2a02e42b6 WatchSource:0}: Error finding container 10828ad6c00bd8c252c80a375c6d02baf1a67becd5a1c64d3355bac2a02e42b6: Status 404 returned error can't find the container with id 10828ad6c00bd8c252c80a375c6d02baf1a67becd5a1c64d3355bac2a02e42b6 Nov 03 16:46:34.813123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:34.813090 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:46:35.547796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.547754 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" event={"ID":"3c842719-571c-4cb7-a442-a88ad7ee53c7","Type":"ContainerStarted","Data":"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431"} Nov 03 16:46:35.548227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.547840 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" event={"ID":"3c842719-571c-4cb7-a442-a88ad7ee53c7","Type":"ContainerStarted","Data":"10828ad6c00bd8c252c80a375c6d02baf1a67becd5a1c64d3355bac2a02e42b6"} Nov 03 16:46:35.548227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.547869 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:35.549591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.549565 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" event={"ID":"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe","Type":"ContainerStarted","Data":"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33"} Nov 03 16:46:35.549726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.549608 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" event={"ID":"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe","Type":"ContainerStarted","Data":"471546825d390c71e0e2fd956bed7101c9a76f5c29a6461538dfe7373cc44cdb"} Nov 03 16:46:35.549726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.549698 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:35.594654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.594598 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" podStartSLOduration=1.594584577 podStartE2EDuration="1.594584577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:35.591579393 +0000 UTC m=+1890.924792962" watchObservedRunningTime="2026-11-03 16:46:35.594584577 +0000 UTC m=+1890.927798145" Nov 03 16:46:35.639239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:35.639164 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" podStartSLOduration=2.639149917 podStartE2EDuration="2.639149917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:35.635185292 +0000 UTC m=+1890.968398861" watchObservedRunningTime="2026-11-03 16:46:35.639149917 +0000 UTC m=+1890.972363485" Nov 03 16:46:36.449329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.448982 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:46:36.588458 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.588412 6001 generic.go:356] "Generic (PLEG): container finished" podID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerID="62fe059155c73051583e03148466b6d7c42acce84957bc433d8adf24e7e55e48" exitCode=0 Nov 03 16:46:36.588880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.588678 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" event={"ID":"61e4fc6a-ca79-4856-bfa8-79a50eb434d9","Type":"ContainerDied","Data":"62fe059155c73051583e03148466b6d7c42acce84957bc433d8adf24e7e55e48"} Nov 03 16:46:36.588880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.588739 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" event={"ID":"61e4fc6a-ca79-4856-bfa8-79a50eb434d9","Type":"ContainerDied","Data":"b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af"} Nov 03 16:46:36.588880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.588764 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b916d46395081c00df8f9b96c65dc0f45bd9443f527593c8f84cf7ec4a8d24af" Nov 03 16:46:36.589896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.589869 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.589998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.589932 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590005 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590025 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590106 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590130 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590152 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590179 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590307 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\") pod \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\" (UID: \"29009892-ebc1-40ca-b2b3-4ccc3fd554aa\") " Nov 03 16:46:36.590721 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590697 6001 generic.go:356] "Generic (PLEG): container finished" podID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerID="557eaf993f5cdf0609790ce0ebc1faf12fcc8e97f72ee67e948b771cd5920031" exitCode=0 Nov 03 16:46:36.590811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.590782 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" event={"ID":"5e653437-91ef-4385-a9fc-4c763318a4a5","Type":"ContainerDied","Data":"557eaf993f5cdf0609790ce0ebc1faf12fcc8e97f72ee67e948b771cd5920031"} Nov 03 16:46:36.591214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.591150 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.591427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.591407 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:36.591534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.591425 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.591660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.591635 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.591723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.591698 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.593252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593227 6001 generic.go:356] "Generic (PLEG): container finished" podID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerID="cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8" exitCode=0 Nov 03 16:46:36.593378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593360 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.593481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593395 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" event={"ID":"29009892-ebc1-40ca-b2b3-4ccc3fd554aa","Type":"ContainerDied","Data":"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8"} Nov 03 16:46:36.593481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593472 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" event={"ID":"29009892-ebc1-40ca-b2b3-4ccc3fd554aa","Type":"ContainerDied","Data":"2f6655bfd28e32765ba4e61c8e3b71d616b756b703a3d4c68d8e080d40caf5ff"} Nov 03 16:46:36.593602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593562 6001 scope.go:122] "RemoveContainer" containerID="cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8" Nov 03 16:46:36.593602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593582 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.593602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.593596 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x" Nov 03 16:46:36.594724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.594697 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "kube-api-access-2jklg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.597280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.597258 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo" pod "29009892-ebc1-40ca-b2b3-4ccc3fd554aa" (UID: "29009892-ebc1-40ca-b2b3-4ccc3fd554aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:36.612749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.612712 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:46:36.617506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.617480 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:46:36.639769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.639736 6001 scope.go:122] "RemoveContainer" containerID="cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8" Nov 03 16:46:36.640377 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:36.640351 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8\": container with ID starting with cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8 not found: ID does not exist" containerID="cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8" Nov 03 16:46:36.640575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.640391 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8"} err="failed to get container status \"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8\": rpc error: code = NotFound desc = could not find container \"cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8\": container with ID starting with cdddb90dcea604af343e45ce175892be2632e2b16454834bf9a8459392e1e0a8 not found: ID does not exist" Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691423 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691533 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691571 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691622 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691684 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691751 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.691883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.691803 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.692356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692329 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692388 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.692489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692428 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.692554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692515 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692573 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692601 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\") pod \"5e653437-91ef-4385-a9fc-4c763318a4a5\" (UID: \"5e653437-91ef-4385-a9fc-4c763318a4a5\") " Nov 03 16:46:36.692709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692655 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692804 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692784 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692875 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692906 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.692950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.692942 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\") pod \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\" (UID: \"61e4fc6a-ca79-4856-bfa8-79a50eb434d9\") " Nov 03 16:46:36.693885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693857 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693889 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693903 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693918 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693931 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693944 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693958 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jklg\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-kube-api-access-2jklg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693970 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.693993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.693983 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29009892-ebc1-40ca-b2b3-4ccc3fd554aa-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.694507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.694481 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.694943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.694919 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.695301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.695276 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.697498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.697437 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:36.698183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.698163 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.698782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.698764 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.700284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.700255 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.700564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.700542 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.705628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.705605 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.705893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.705868 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.707907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.707879 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.708096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.708078 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.708301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.708274 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:36.708705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.708682 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:36.710385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.710345 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "kube-api-access-ws7p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.710729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.710567 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo" pod "5e653437-91ef-4385-a9fc-4c763318a4a5" (UID: "5e653437-91ef-4385-a9fc-4c763318a4a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:36.753854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.753813 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:36.754714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.754683 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj" pod "61e4fc6a-ca79-4856-bfa8-79a50eb434d9" (UID: "61e4fc6a-ca79-4856-bfa8-79a50eb434d9"). InnerVolumeSpecName "kube-api-access-q5pvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:36.795038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795000 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pvj\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-kube-api-access-q5pvj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795035 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795049 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795064 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795078 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795091 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795104 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795118 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795132 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795145 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795159 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795171 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795184 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795196 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws7p2\" (UniqueName: \"kubernetes.io/projected/5e653437-91ef-4385-a9fc-4c763318a4a5-kube-api-access-ws7p2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795208 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795221 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61e4fc6a-ca79-4856-bfa8-79a50eb434d9-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795775 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795235 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e653437-91ef-4385-a9fc-4c763318a4a5-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.795775 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.795251 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e653437-91ef-4385-a9fc-4c763318a4a5-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:36.937136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.937097 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:46:36.942852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:36.942789 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-pt76x"] Nov 03 16:46:37.583797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.583757 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:46:37.583989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.583951 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" containerID="cri-o://d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046" gracePeriod=30 Nov 03 16:46:37.597394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.597360 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:46:37.597911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.597697 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" containerID="cri-o://74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343" gracePeriod=30 Nov 03 16:46:37.601817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.601788 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:46:37.602037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.602009 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" containerID="cri-o://8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6" gracePeriod=30 Nov 03 16:46:37.604508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.604482 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" Nov 03 16:46:37.604639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.604481 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs" event={"ID":"5e653437-91ef-4385-a9fc-4c763318a4a5","Type":"ContainerDied","Data":"bd1e3bebadd3ae9460dafd729675d6cef9b17f089ca38681160dfe256642c8be"} Nov 03 16:46:37.604745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.604725 6001 scope.go:122] "RemoveContainer" containerID="557eaf993f5cdf0609790ce0ebc1faf12fcc8e97f72ee67e948b771cd5920031" Nov 03 16:46:37.606205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.606180 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b" Nov 03 16:46:37.672605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.672518 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:46:37.676036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.676008 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sgrxs"] Nov 03 16:46:37.760970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.760939 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:46:37.764815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:37.764786 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-4rn5b"] Nov 03 16:46:38.791903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:38.791866 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" path="/var/lib/kubelet/pods/29009892-ebc1-40ca-b2b3-4ccc3fd554aa/volumes" Nov 03 16:46:38.792431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:38.792416 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" path="/var/lib/kubelet/pods/5e653437-91ef-4385-a9fc-4c763318a4a5/volumes" Nov 03 16:46:38.792891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:38.792876 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" path="/var/lib/kubelet/pods/61e4fc6a-ca79-4856-bfa8-79a50eb434d9/volumes" Nov 03 16:46:40.586751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:40.585427 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:46:40.588488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:40.588462 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:46:42.936469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:42.934383 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:46:42.938077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:42.938052 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:46:42.999392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:42.998184 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:46:43.022621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022592 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.022793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022628 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.022793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022648 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.022793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022738 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.022793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022770 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.023032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022804 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022915 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.023032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.022859 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023061 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.023156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023080 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.023156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023099 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023172 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023201 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023232 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.023339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023265 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.023339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023292 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.023339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023321 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.023427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023360 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.023507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.023507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023489 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:43.023629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023526 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.024006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.023975 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.024126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024089 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.024197 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024161 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.024256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024207 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.024256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024233 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.024360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024300 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.024360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\") pod \"510901d2-e32c-4348-b5d3-d6549bf76458\" (UID: \"510901d2-e32c-4348-b5d3-d6549bf76458\") " Nov 03 16:46:43.024543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024379 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\") pod \"48dcf133-89d9-4315-b38c-268f6740f920\" (UID: \"48dcf133-89d9-4315-b38c-268f6740f920\") " Nov 03 16:46:43.024543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.024414 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\") pod \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\" (UID: \"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6\") " Nov 03 16:46:43.025219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.025193 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.025481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.025419 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.025847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.025818 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:43.026132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.026113 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.026207 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.026139 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.026355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.026334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:43.026887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.026672 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:43.026887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.026806 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:43.027208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027031 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.027273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027218 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027508 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027542 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027748 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027755 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027875 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027872 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.027959 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028089 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028101 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028197 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028305 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028468 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "kube-api-access-lc69x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.028930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.028903 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "kube-api-access-bdhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.029688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.029535 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:43.030064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.030042 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token" pod "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" (UID: "b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.030345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.030325 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2" pod "510901d2-e32c-4348-b5d3-d6549bf76458" (UID: "510901d2-e32c-4348-b5d3-d6549bf76458"). InnerVolumeSpecName "kube-api-access-2qkk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.031368 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.031334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token" pod "48dcf133-89d9-4315-b38c-268f6740f920" (UID: "48dcf133-89d9-4315-b38c-268f6740f920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:43.127488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127435 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127488 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/510901d2-e32c-4348-b5d3-d6549bf76458-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127501 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127514 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127527 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127538 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127549 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127562 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkk2\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-kube-api-access-2qkk2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127593 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127605 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127617 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc69x\" (UniqueName: \"kubernetes.io/projected/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-kube-api-access-lc69x\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127628 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127642 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127653 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127665 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48dcf133-89d9-4315-b38c-268f6740f920-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127677 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127687 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127699 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48dcf133-89d9-4315-b38c-268f6740f920-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127713 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdhhs\" (UniqueName: \"kubernetes.io/projected/48dcf133-89d9-4315-b38c-268f6740f920-kube-api-access-bdhhs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127727 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.127734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127741 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48dcf133-89d9-4315-b38c-268f6740f920-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.128311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127754 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/510901d2-e32c-4348-b5d3-d6549bf76458-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.128311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127767 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/510901d2-e32c-4348-b5d3-d6549bf76458-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.128311 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.127780 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/510901d2-e32c-4348-b5d3-d6549bf76458-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:43.665354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.665310 6001 generic.go:356] "Generic (PLEG): container finished" podID="510901d2-e32c-4348-b5d3-d6549bf76458" containerID="8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6" exitCode=0 Nov 03 16:46:43.665599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.665366 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" Nov 03 16:46:43.665599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.665397 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" event={"ID":"510901d2-e32c-4348-b5d3-d6549bf76458","Type":"ContainerDied","Data":"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6"} Nov 03 16:46:43.665599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.665498 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq" event={"ID":"510901d2-e32c-4348-b5d3-d6549bf76458","Type":"ContainerDied","Data":"37f7ca8bff0ea720f727f7463cce2e2fe88f5a874ace8be29e47f88e0a000097"} Nov 03 16:46:43.665599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.665525 6001 scope.go:122] "RemoveContainer" containerID="8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6" Nov 03 16:46:43.667006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.666973 6001 generic.go:356] "Generic (PLEG): container finished" podID="48dcf133-89d9-4315-b38c-268f6740f920" containerID="74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343" exitCode=0 Nov 03 16:46:43.667135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.667023 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" Nov 03 16:46:43.667135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.667060 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" event={"ID":"48dcf133-89d9-4315-b38c-268f6740f920","Type":"ContainerDied","Data":"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343"} Nov 03 16:46:43.667135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.667116 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm" event={"ID":"48dcf133-89d9-4315-b38c-268f6740f920","Type":"ContainerDied","Data":"0a35d4399d31e5416b70189ee055fac7ce535388cff763e0b43c79738c60c535"} Nov 03 16:46:43.668851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.668820 6001 generic.go:356] "Generic (PLEG): container finished" podID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerID="d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046" exitCode=0 Nov 03 16:46:43.668970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.668875 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" Nov 03 16:46:43.668970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.668946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" event={"ID":"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6","Type":"ContainerDied","Data":"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046"} Nov 03 16:46:43.669086 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.668987 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g" event={"ID":"b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6","Type":"ContainerDied","Data":"89489e489bae75bb1a82f816c373a285c7239e6fe5c23a3d7867e818fc8f3ec2"} Nov 03 16:46:43.678351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.678132 6001 scope.go:122] "RemoveContainer" containerID="8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6" Nov 03 16:46:43.678638 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:43.678610 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6\": container with ID starting with 8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6 not found: ID does not exist" containerID="8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6" Nov 03 16:46:43.678753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.678648 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6"} err="failed to get container status \"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6\": rpc error: code = NotFound desc = could not find container \"8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6\": container with ID starting with 8f7dae0929fa8fcba2d6f972d204ef762405a208ff2f39cb53ed7277fb1bdcc6 not found: ID does not exist" Nov 03 16:46:43.678753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.678670 6001 scope.go:122] "RemoveContainer" containerID="74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343" Nov 03 16:46:43.690430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.690405 6001 scope.go:122] "RemoveContainer" containerID="74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343" Nov 03 16:46:43.690773 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:43.690751 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343\": container with ID starting with 74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343 not found: ID does not exist" containerID="74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343" Nov 03 16:46:43.690831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.690781 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343"} err="failed to get container status \"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343\": rpc error: code = NotFound desc = could not find container \"74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343\": container with ID starting with 74d8c961a2dac104b56f12ddb199f2a05cb026177dee18400e50ebc6f8eae343 not found: ID does not exist" Nov 03 16:46:43.690831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.690800 6001 scope.go:122] "RemoveContainer" containerID="d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046" Nov 03 16:46:43.701254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.701234 6001 scope.go:122] "RemoveContainer" containerID="d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046" Nov 03 16:46:43.701589 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:43.701570 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046\": container with ID starting with d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046 not found: ID does not exist" containerID="d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046" Nov 03 16:46:43.701646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.701597 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046"} err="failed to get container status \"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046\": rpc error: code = NotFound desc = could not find container \"d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046\": container with ID starting with d9b2c21aca9983524595825fbfb4165603a52ce8ff8ab1fc31a0fc10be6a7046 not found: ID does not exist" Nov 03 16:46:43.754201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.754169 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:46:43.766691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.766658 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-d58sq"] Nov 03 16:46:43.804584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.804546 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:46:43.807702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.807672 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-zlk4g"] Nov 03 16:46:43.845631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.845600 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:46:43.852132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:43.852092 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-kvjbm"] Nov 03 16:46:44.498370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.498324 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:46:44.499223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499197 6001 cpu_manager.go:414] "containerMap: removing container" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" Nov 03 16:46:44.499223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499223 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499237 6001 cpu_manager.go:414] "containerMap: removing container" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499245 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499325 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499333 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499351 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499359 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499376 6001 cpu_manager.go:414] "containerMap: removing container" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499384 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" Nov 03 16:46:44.499398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499397 6001 cpu_manager.go:414] "containerMap: removing container" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499405 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499600 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="29009892-ebc1-40ca-b2b3-4ccc3fd554aa" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499619 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="48dcf133-89d9-4315-b38c-268f6740f920" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499630 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="61e4fc6a-ca79-4856-bfa8-79a50eb434d9" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499657 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499669 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e653437-91ef-4385-a9fc-4c763318a4a5" containerName="istio-proxy" Nov 03 16:46:44.499854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.499679 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" containerName="istio-proxy" Nov 03 16:46:44.503695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.503672 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.540615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.540577 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:46:44.547575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.547535 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9947ab-openshift-default-dockercfg-9wmhj\"" Nov 03 16:46:44.550429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.550397 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.550682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.550657 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.550740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.550720 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.550838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.550814 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sg4s\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.550887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.550868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.551041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.551022 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.551084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.551068 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.551171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.551151 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.551220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.551208 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652532 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652587 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652622 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652653 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652735 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652854 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652934 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sg4s\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.652983 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.653886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.653515 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.657664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.656927 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.657664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.657188 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.657664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.657623 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.658416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.658380 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.667345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.667282 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.669890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.667978 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.685542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.685497 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.686962 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.686934 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sg4s\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\") pod \"e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:44.792638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.792557 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48dcf133-89d9-4315-b38c-268f6740f920" path="/var/lib/kubelet/pods/48dcf133-89d9-4315-b38c-268f6740f920/volumes" Nov 03 16:46:44.793105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.793091 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510901d2-e32c-4348-b5d3-d6549bf76458" path="/var/lib/kubelet/pods/510901d2-e32c-4348-b5d3-d6549bf76458/volumes" Nov 03 16:46:44.793563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.793545 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6" path="/var/lib/kubelet/pods/b7eb9b6a-4ed5-4d64-af4b-a67a9cf151f6/volumes" Nov 03 16:46:44.819045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:44.819001 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:45.141729 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:45.141699 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440ab9e3_4d80_4ab2_a596_c6e0e1879499.slice/crio-4c33c71655251d8f614bc5788c000206e345bb82757bdf7b1d6e135a62ff52ad WatchSource:0}: Error finding container 4c33c71655251d8f614bc5788c000206e345bb82757bdf7b1d6e135a62ff52ad: Status 404 returned error can't find the container with id 4c33c71655251d8f614bc5788c000206e345bb82757bdf7b1d6e135a62ff52ad Nov 03 16:46:45.141933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.141911 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:46:45.144700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.144664 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:46:45.144819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.144746 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:46:45.144819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.144793 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:46:45.693031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.692985 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerStarted","Data":"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b"} Nov 03 16:46:45.693438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.693077 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerStarted","Data":"4c33c71655251d8f614bc5788c000206e345bb82757bdf7b1d6e135a62ff52ad"} Nov 03 16:46:45.780466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.776583 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podStartSLOduration=1.776564188 podStartE2EDuration="1.776564188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:45.758691912 +0000 UTC m=+1901.091905480" watchObservedRunningTime="2026-11-03 16:46:45.776564188 +0000 UTC m=+1901.109777757" Nov 03 16:46:45.819984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:45.819946 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:46.353936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.353840 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:46:46.357455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.357415 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.379298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.379264 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqrqp\" (UniqueName: \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.379554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.379533 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.381175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.381141 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-87b132\"" Nov 03 16:46:46.381286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.381272 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-87b132-dockercfg-42r72\"" Nov 03 16:46:46.386480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.384166 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:46:46.483496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.468463 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:46:46.483496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.482845 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.483496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.483035 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqrqp\" (UniqueName: \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.485264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.484345 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.513061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.505235 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.513061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.509051 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqrqp\" (UniqueName: \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\") pod \"payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.513061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.512949 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:46:46.583933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.583897 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.584112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.584075 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgpcf\" (UniqueName: \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.595810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.595774 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:46:46.596399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.596375 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:46:46.672797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.672581 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:46.686253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.686216 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.686855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.686807 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgpcf\" (UniqueName: \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.688587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.688561 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.724109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.724042 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgpcf\" (UniqueName: \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\") pod \"payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:46.827465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.827423 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:46.860827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:46.860790 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:47.008011 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:47.007970 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5411db87_17cc_4d9e_ac24_9f9fbda98056.slice/crio-cec33d36f39ed456f5073ca3b9ed5a546254b03991f4b1d11cbda5225d18bf84 WatchSource:0}: Error finding container cec33d36f39ed456f5073ca3b9ed5a546254b03991f4b1d11cbda5225d18bf84: Status 404 returned error can't find the container with id cec33d36f39ed456f5073ca3b9ed5a546254b03991f4b1d11cbda5225d18bf84 Nov 03 16:46:47.009258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.009229 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:46:47.222677 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:46:47.222642 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025b982b_9cee_4ce1_84fe_7d5a04632f2d.slice/crio-6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8 WatchSource:0}: Error finding container 6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8: Status 404 returned error can't find the container with id 6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8 Nov 03 16:46:47.223135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.223114 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:46:47.713222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.713183 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" event={"ID":"025b982b-9cee-4ce1-84fe-7d5a04632f2d","Type":"ContainerStarted","Data":"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4"} Nov 03 16:46:47.713403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.713240 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" event={"ID":"025b982b-9cee-4ce1-84fe-7d5a04632f2d","Type":"ContainerStarted","Data":"6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8"} Nov 03 16:46:47.713509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.713401 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:47.715296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.715266 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" event={"ID":"5411db87-17cc-4d9e-ac24-9f9fbda98056","Type":"ContainerStarted","Data":"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230"} Nov 03 16:46:47.715427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.715319 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" event={"ID":"5411db87-17cc-4d9e-ac24-9f9fbda98056","Type":"ContainerStarted","Data":"cec33d36f39ed456f5073ca3b9ed5a546254b03991f4b1d11cbda5225d18bf84"} Nov 03 16:46:47.715427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.715349 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:47.715681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.715663 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:47.716932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.716911 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:46:47.765114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.765054 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" podStartSLOduration=1.7650405980000001 podStartE2EDuration="1.765040598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:47.760512293 +0000 UTC m=+1903.093725863" watchObservedRunningTime="2026-11-03 16:46:47.765040598 +0000 UTC m=+1903.098254175" Nov 03 16:46:47.907747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:47.907226 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" podStartSLOduration=1.9072047049999998 podStartE2EDuration="1.907204705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:47.902697481 +0000 UTC m=+1903.235911061" watchObservedRunningTime="2026-11-03 16:46:47.907204705 +0000 UTC m=+1903.240418275" Nov 03 16:46:50.843106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.843077 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:46:50.847727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.847710 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:50.867271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.867244 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:46:50.876391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.876367 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9947ab\"" Nov 03 16:46:50.876539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.876398 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9947ab-dockercfg-xw4rx\"" Nov 03 16:46:50.921599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.921538 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:46:50.927204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.927174 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:50.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.953811 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:46:50.975638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.964213 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwrc\" (UniqueName: \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:50.975638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:50.964314 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.067498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.065274 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.067498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.065478 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.067498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.065655 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwrc\" (UniqueName: \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.067498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.065762 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqv9\" (UniqueName: \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.067498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.066222 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.090117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.090089 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwrc\" (UniqueName: \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.163207 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.163108 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.167174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.166835 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqv9\" (UniqueName: \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.167174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.166979 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.167673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.167650 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.191158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.191124 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqv9\" (UniqueName: \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\") pod \"payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.255510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.255472 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.518810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.518766 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:46:51.621097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.621062 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:46:51.749071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.749033 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" event={"ID":"75dc1547-aabb-43ee-91a4-72726183a03a","Type":"ContainerStarted","Data":"0e52f41961652c6ef71fed88522e93e5b5809bf33fdaee5b09774dc29356a39d"} Nov 03 16:46:51.749265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.749098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" event={"ID":"75dc1547-aabb-43ee-91a4-72726183a03a","Type":"ContainerStarted","Data":"d5fdde38fe8362ec4105ddf1e48a78781668bd25626199da11619773f2762c0c"} Nov 03 16:46:51.749265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.749124 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:46:51.750990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.750959 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" event={"ID":"dbe8976e-5513-4196-abbf-81035edeac65","Type":"ContainerStarted","Data":"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367"} Nov 03 16:46:51.751134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.751008 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" event={"ID":"dbe8976e-5513-4196-abbf-81035edeac65","Type":"ContainerStarted","Data":"7a462b9611005cb1180c7c3a65c685897481635d7faa48ff2c9a6f052165011a"} Nov 03 16:46:51.751134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.751033 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:46:51.798694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.798588 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" podStartSLOduration=1.79857429 podStartE2EDuration="1.79857429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:51.797391574 +0000 UTC m=+1907.130605144" watchObservedRunningTime="2026-11-03 16:46:51.79857429 +0000 UTC m=+1907.131787940" Nov 03 16:46:51.845639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:51.845577 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" podStartSLOduration=1.845560621 podStartE2EDuration="1.845560621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:51.842757035 +0000 UTC m=+1907.175970615" watchObservedRunningTime="2026-11-03 16:46:51.845560621 +0000 UTC m=+1907.178774184" Nov 03 16:46:52.203852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:52.203768 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:46:52.253637 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:52.253590 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:46:52.270653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:52.270618 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:46:57.204762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.204720 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" podUID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" containerName="payload-processing" containerID="cri-o://8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825" gracePeriod=25 Nov 03 16:46:57.271806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.271747 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" podUID="4ad3498f-715c-4be0-83a2-33800ea65757" containerName="payload-pre-processing" containerID="cri-o://8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347" gracePeriod=25 Nov 03 16:46:57.557056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.557031 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:46:57.598682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.598650 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:46:57.686307 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.686272 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\" (UniqueName: \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\") pod \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " Nov 03 16:46:57.686546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.686525 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\") pod \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\" (UID: \"d45c94a8-b00e-4509-8c42-5d2eb3b50346\") " Nov 03 16:46:57.686901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.686865 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume" pod "d45c94a8-b00e-4509-8c42-5d2eb3b50346" (UID: "d45c94a8-b00e-4509-8c42-5d2eb3b50346"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:57.687180 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.687165 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d45c94a8-b00e-4509-8c42-5d2eb3b50346-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:57.689064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.689041 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5" pod "d45c94a8-b00e-4509-8c42-5d2eb3b50346" (UID: "d45c94a8-b00e-4509-8c42-5d2eb3b50346"). InnerVolumeSpecName "kube-api-access-pt8l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:57.788056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.787974 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\" (UniqueName: \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\") pod \"4ad3498f-715c-4be0-83a2-33800ea65757\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " Nov 03 16:46:57.788206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.788060 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\") pod \"4ad3498f-715c-4be0-83a2-33800ea65757\" (UID: \"4ad3498f-715c-4be0-83a2-33800ea65757\") " Nov 03 16:46:57.788479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.788433 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume" pod "4ad3498f-715c-4be0-83a2-33800ea65757" (UID: "4ad3498f-715c-4be0-83a2-33800ea65757"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:57.788609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.788581 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt8l5\" (UniqueName: \"kubernetes.io/projected/d45c94a8-b00e-4509-8c42-5d2eb3b50346-kube-api-access-pt8l5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:57.788609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.788595 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad3498f-715c-4be0-83a2-33800ea65757-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:57.790103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.790077 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h" pod "4ad3498f-715c-4be0-83a2-33800ea65757" (UID: "4ad3498f-715c-4be0-83a2-33800ea65757"). InnerVolumeSpecName "kube-api-access-txh9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:57.815261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.815223 6001 generic.go:356] "Generic (PLEG): container finished" podID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" containerID="8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825" exitCode=0 Nov 03 16:46:57.815396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.815291 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" Nov 03 16:46:57.815396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.815309 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" event={"ID":"d45c94a8-b00e-4509-8c42-5d2eb3b50346","Type":"ContainerDied","Data":"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825"} Nov 03 16:46:57.815396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.815376 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s" event={"ID":"d45c94a8-b00e-4509-8c42-5d2eb3b50346","Type":"ContainerDied","Data":"ef1db34d1a9d2dff6cd535a098e8218e010ac15d9e6184642c11a2b79412fcd9"} Nov 03 16:46:57.815590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.815403 6001 scope.go:122] "RemoveContainer" containerID="8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825" Nov 03 16:46:57.817105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.817067 6001 generic.go:356] "Generic (PLEG): container finished" podID="4ad3498f-715c-4be0-83a2-33800ea65757" containerID="8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347" exitCode=0 Nov 03 16:46:57.817223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.817122 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" Nov 03 16:46:57.817223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.817134 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" event={"ID":"4ad3498f-715c-4be0-83a2-33800ea65757","Type":"ContainerDied","Data":"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347"} Nov 03 16:46:57.817223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.817183 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv" event={"ID":"4ad3498f-715c-4be0-83a2-33800ea65757","Type":"ContainerDied","Data":"2db6c50eb14f4266b688b09ba1ddb240fbdee83a2e23648cdf44a9c54aba8aee"} Nov 03 16:46:57.827459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.827425 6001 scope.go:122] "RemoveContainer" containerID="8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825" Nov 03 16:46:57.827757 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:57.827736 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825\": container with ID starting with 8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825 not found: ID does not exist" containerID="8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825" Nov 03 16:46:57.827823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.827765 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825"} err="failed to get container status \"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825\": rpc error: code = NotFound desc = could not find container \"8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825\": container with ID starting with 8b78ccab9a842915864c670936d4a0fdc8a12d46a365c34f9a4880e969bb7825 not found: ID does not exist" Nov 03 16:46:57.827823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.827784 6001 scope.go:122] "RemoveContainer" containerID="8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347" Nov 03 16:46:57.838030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.838013 6001 scope.go:122] "RemoveContainer" containerID="8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347" Nov 03 16:46:57.838284 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:46:57.838265 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347\": container with ID starting with 8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347 not found: ID does not exist" containerID="8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347" Nov 03 16:46:57.838327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.838291 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347"} err="failed to get container status \"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347\": rpc error: code = NotFound desc = could not find container \"8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347\": container with ID starting with 8c60e5a81bd7b0c87f6b452a5158bb7ebe0e601dd878b6ab6a01972710e81347 not found: ID does not exist" Nov 03 16:46:57.870588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.870545 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:46:57.875469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.875429 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-6cd6985d7f-6zsbv"] Nov 03 16:46:57.890759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.890732 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txh9h\" (UniqueName: \"kubernetes.io/projected/4ad3498f-715c-4be0-83a2-33800ea65757-kube-api-access-txh9h\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:46:57.910950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.910921 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:46:57.913925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:57.913907 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-64695bbbc9-vh92s"] Nov 03 16:46:58.726685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:58.726652 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:46:58.727390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:58.727367 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:46:58.796894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:58.796858 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad3498f-715c-4be0-83a2-33800ea65757" path="/var/lib/kubelet/pods/4ad3498f-715c-4be0-83a2-33800ea65757/volumes" Nov 03 16:46:58.797562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:46:58.797540 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" path="/var/lib/kubelet/pods/d45c94a8-b00e-4509-8c42-5d2eb3b50346/volumes" Nov 03 16:47:02.376963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.376928 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:47:02.377422 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.377150 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" podUID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" containerName="payload-processing" containerID="cri-o://fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652" gracePeriod=30 Nov 03 16:47:02.753189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.747971 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:47:02.763192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.763073 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:47:02.772428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.772399 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:47:02.866583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.866538 6001 generic.go:356] "Generic (PLEG): container finished" podID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" containerID="fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652" exitCode=0 Nov 03 16:47:02.866796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.866611 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" Nov 03 16:47:02.866796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.866637 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" event={"ID":"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f","Type":"ContainerDied","Data":"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652"} Nov 03 16:47:02.866796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.866722 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p" event={"ID":"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f","Type":"ContainerDied","Data":"42247c809e8b5b5e1efc4fccda4600bfe56d49844a658b938dbd48b5d974ad4c"} Nov 03 16:47:02.866796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.866755 6001 scope.go:122] "RemoveContainer" containerID="fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652" Nov 03 16:47:02.889962 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.889936 6001 scope.go:122] "RemoveContainer" containerID="fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652" Nov 03 16:47:02.890353 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:02.890324 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652\": container with ID starting with fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652 not found: ID does not exist" containerID="fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652" Nov 03 16:47:02.890493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.890365 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652"} err="failed to get container status \"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652\": rpc error: code = NotFound desc = could not find container \"fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652\": container with ID starting with fdd2a4b0d603663acf67e71eff2f4fb1f72a5a7f1c8e496d3d29ae1f314d3652 not found: ID does not exist" Nov 03 16:47:02.921732 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.921633 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\") pod \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " Nov 03 16:47:02.922227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.922195 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\" (UniqueName: \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\") pod \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\" (UID: \"e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f\") " Nov 03 16:47:02.922613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.922042 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume" pod "e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" (UID: "e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:02.923089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.923063 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:02.925668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:02.925634 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp" pod "e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" (UID: "e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f"). InnerVolumeSpecName "kube-api-access-w7jbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:03.024236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:03.024202 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7jbp\" (UniqueName: \"kubernetes.io/projected/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f-kube-api-access-w7jbp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.215907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:03.215870 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:47:03.224903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:03.224720 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-6f96f65c6b-8pm8p"] Nov 03 16:47:04.802781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:04.802742 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" path="/var/lib/kubelet/pods/e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f/volumes" Nov 03 16:47:08.620892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.620858 6001 scope.go:122] "RemoveContainer" containerID="003b55d780fabaed1c3509bbeb368f1c10b76517ac914bd968dff9174c4cda4e" Nov 03 16:47:08.631016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.630993 6001 scope.go:122] "RemoveContainer" containerID="224e11c854e7a83d6c756e2fc66179f8fa953471c17fa69554dae0f81666f439" Nov 03 16:47:08.640173 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.640145 6001 scope.go:122] "RemoveContainer" containerID="62fe059155c73051583e03148466b6d7c42acce84957bc433d8adf24e7e55e48" Nov 03 16:47:08.649277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.649258 6001 scope.go:122] "RemoveContainer" containerID="a315792ee2d90441a66803a2263eac2a1bbc27e07dbb2227287928ed1483a3e5" Nov 03 16:47:08.658578 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.658551 6001 scope.go:122] "RemoveContainer" containerID="9dbf7863d91193dd6a3130f528f84a08622877629ffcd42d8baf51dacce7b40d" Nov 03 16:47:08.667187 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.667171 6001 scope.go:122] "RemoveContainer" containerID="bbf483a18730f7b1b4818b034520b9dac1607e6bf563b20bc416521e26eb4f74" Nov 03 16:47:08.675708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:08.675685 6001 scope.go:122] "RemoveContainer" containerID="5e2fc06388743325db6dc6df05728548aa5171f68a3b71490086c20361356c49" Nov 03 16:47:12.690692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.690655 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:47:12.692983 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.692949 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" containerName="payload-processing" Nov 03 16:47:12.692983 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.692982 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" containerName="payload-processing" Nov 03 16:47:12.693220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693013 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4ad3498f-715c-4be0-83a2-33800ea65757" containerName="payload-pre-processing" Nov 03 16:47:12.693220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693022 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad3498f-715c-4be0-83a2-33800ea65757" containerName="payload-pre-processing" Nov 03 16:47:12.693220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693039 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" containerName="payload-processing" Nov 03 16:47:12.693220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693052 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" containerName="payload-processing" Nov 03 16:47:12.693432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693328 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d2ddf3-2df8-4bc3-93a3-ede6e5880e4f" containerName="payload-processing" Nov 03 16:47:12.693432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693345 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad3498f-715c-4be0-83a2-33800ea65757" containerName="payload-pre-processing" Nov 03 16:47:12.693432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.693356 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45c94a8-b00e-4509-8c42-5d2eb3b50346" containerName="payload-processing" Nov 03 16:47:12.700786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.700761 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.731478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.730718 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:47:12.743015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.742860 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-624649\"" Nov 03 16:47:12.743015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.742943 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-624649-dockercfg-hb8x6\"" Nov 03 16:47:12.758608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.758571 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbsk\" (UniqueName: \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.758812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.758647 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.859867 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.859823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.860163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.860137 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbsk\" (UniqueName: \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.860679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.860651 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.870234 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.870197 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:47:12.876392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.876358 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:12.890045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.890007 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbsk\" (UniqueName: \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\") pod \"payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:12.895699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.895669 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:47:12.961624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.961532 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4bh\" (UniqueName: \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:12.961624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:12.961589 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.020208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.020170 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:13.063287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.063241 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.063556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.063537 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4bh\" (UniqueName: \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.064066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.064042 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.088534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.088467 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4bh\" (UniqueName: \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\") pod \"payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.202748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.202709 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.334899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.334874 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:47:13.509628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.509595 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:47:13.510277 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:13.510246 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d91eb8_f9c5_4106_8b8a_11023a9a7a94.slice/crio-4b560ffe5ab8c3eaba15c3349e8e1928a2132d5e890e575a1c27c1b6540a050f WatchSource:0}: Error finding container 4b560ffe5ab8c3eaba15c3349e8e1928a2132d5e890e575a1c27c1b6540a050f: Status 404 returned error can't find the container with id 4b560ffe5ab8c3eaba15c3349e8e1928a2132d5e890e575a1c27c1b6540a050f Nov 03 16:47:13.949069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.948971 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" event={"ID":"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94","Type":"ContainerStarted","Data":"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498"} Nov 03 16:47:13.949069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.949038 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" event={"ID":"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94","Type":"ContainerStarted","Data":"4b560ffe5ab8c3eaba15c3349e8e1928a2132d5e890e575a1c27c1b6540a050f"} Nov 03 16:47:13.949660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.949199 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:13.951105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.951075 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" event={"ID":"5f755daf-7dc0-4009-b803-8813f631cad0","Type":"ContainerStarted","Data":"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533"} Nov 03 16:47:13.951241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.951128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" event={"ID":"5f755daf-7dc0-4009-b803-8813f631cad0","Type":"ContainerStarted","Data":"2e8eeec555d55aae6f2a46369b39a33f8234b305d3efc0884130995d1b8aabc0"} Nov 03 16:47:13.951241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:13.951199 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:14.000860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:14.000787 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" podStartSLOduration=2.000766009 podStartE2EDuration="2.000766009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:13.997704456 +0000 UTC m=+1929.330918051" watchObservedRunningTime="2026-11-03 16:47:14.000766009 +0000 UTC m=+1929.333979581" Nov 03 16:47:14.045378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:14.045306 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" podStartSLOduration=2.0452885 podStartE2EDuration="2.0452885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:14.042839625 +0000 UTC m=+1929.376053195" watchObservedRunningTime="2026-11-03 16:47:14.0452885 +0000 UTC m=+1929.378502076" Nov 03 16:47:21.908638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:21.908604 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:47:21.943092 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:21.943052 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:47:22.252650 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:22.252612 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:47:24.040809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.040774 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t"] Nov 03 16:47:24.053704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.053668 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3752ff2-bhn6t"] Nov 03 16:47:24.592842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.592808 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:47:24.593053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.593016 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" podUID="3c842719-571c-4cb7-a442-a88ad7ee53c7" containerName="payload-processing" containerID="cri-o://da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431" gracePeriod=30 Nov 03 16:47:24.612845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.612806 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:47:24.613094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.613064 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" podUID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" containerName="payload-pre-processing" containerID="cri-o://54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33" gracePeriod=30 Nov 03 16:47:24.808177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.807368 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bc1bf0-b0e6-499a-aff8-068958ac7538" path="/var/lib/kubelet/pods/e8bc1bf0-b0e6-499a-aff8-068958ac7538/volumes" Nov 03 16:47:24.923551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.923524 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:47:24.961530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.961500 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:47:24.961677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:24.961594 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:47:25.011470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.008303 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:47:25.057122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.057072 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\") pod \"3c842719-571c-4cb7-a442-a88ad7ee53c7\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " Nov 03 16:47:25.057753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.057229 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\" (UniqueName: \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\") pod \"3c842719-571c-4cb7-a442-a88ad7ee53c7\" (UID: \"3c842719-571c-4cb7-a442-a88ad7ee53c7\") " Nov 03 16:47:25.057753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.057498 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume" pod "3c842719-571c-4cb7-a442-a88ad7ee53c7" (UID: "3c842719-571c-4cb7-a442-a88ad7ee53c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:25.058834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.058139 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c842719-571c-4cb7-a442-a88ad7ee53c7-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.078465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.078415 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr" pod "3c842719-571c-4cb7-a442-a88ad7ee53c7" (UID: "3c842719-571c-4cb7-a442-a88ad7ee53c7"). InnerVolumeSpecName "kube-api-access-ptxnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:25.078787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.078746 6001 generic.go:356] "Generic (PLEG): container finished" podID="3c842719-571c-4cb7-a442-a88ad7ee53c7" containerID="da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431" exitCode=0 Nov 03 16:47:25.078948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.078859 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" event={"ID":"3c842719-571c-4cb7-a442-a88ad7ee53c7","Type":"ContainerDied","Data":"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431"} Nov 03 16:47:25.078948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.078916 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" event={"ID":"3c842719-571c-4cb7-a442-a88ad7ee53c7","Type":"ContainerDied","Data":"10828ad6c00bd8c252c80a375c6d02baf1a67becd5a1c64d3355bac2a02e42b6"} Nov 03 16:47:25.078948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.078946 6001 scope.go:122] "RemoveContainer" containerID="da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431" Nov 03 16:47:25.079141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.079125 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6" Nov 03 16:47:25.114714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.114685 6001 generic.go:356] "Generic (PLEG): container finished" podID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" containerID="54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33" exitCode=0 Nov 03 16:47:25.114996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.114972 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" event={"ID":"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe","Type":"ContainerDied","Data":"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33"} Nov 03 16:47:25.115060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.115033 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" event={"ID":"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe","Type":"ContainerDied","Data":"471546825d390c71e0e2fd956bed7101c9a76f5c29a6461538dfe7373cc44cdb"} Nov 03 16:47:25.115131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.115117 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5" Nov 03 16:47:25.130747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.130718 6001 scope.go:122] "RemoveContainer" containerID="da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431" Nov 03 16:47:25.131147 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:25.131119 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431\": container with ID starting with da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431 not found: ID does not exist" containerID="da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431" Nov 03 16:47:25.131307 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.131160 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431"} err="failed to get container status \"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431\": rpc error: code = NotFound desc = could not find container \"da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431\": container with ID starting with da8b3065735edc5507a8d8e5af522f0737a806acfb2989de22a98381dc6cd431 not found: ID does not exist" Nov 03 16:47:25.131307 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.131188 6001 scope.go:122] "RemoveContainer" containerID="54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33" Nov 03 16:47:25.148561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.148535 6001 scope.go:122] "RemoveContainer" containerID="54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33" Nov 03 16:47:25.149009 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:25.148984 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33\": container with ID starting with 54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33 not found: ID does not exist" containerID="54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33" Nov 03 16:47:25.149106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.149020 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33"} err="failed to get container status \"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33\": rpc error: code = NotFound desc = could not find container \"54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33\": container with ID starting with 54eaac23eb22994847712b41c301b722e679fb1711885d704866db1e5b2b8b33 not found: ID does not exist" Nov 03 16:47:25.159657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.159625 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\" (UniqueName: \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\") pod \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " Nov 03 16:47:25.159866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.159848 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\") pod \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\" (UID: \"f1a6628a-7e6e-46c3-b076-b98fb9dfdefe\") " Nov 03 16:47:25.160303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.160270 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume" pod "f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" (UID: "f1a6628a-7e6e-46c3-b076-b98fb9dfdefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:25.160530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.160513 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.160599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.160534 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptxnr\" (UniqueName: \"kubernetes.io/projected/3c842719-571c-4cb7-a442-a88ad7ee53c7-kube-api-access-ptxnr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.162393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.162366 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2" pod "f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" (UID: "f1a6628a-7e6e-46c3-b076-b98fb9dfdefe"). InnerVolumeSpecName "kube-api-access-k9wr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:25.210921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.210885 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:47:25.220545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.220511 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-53bd50-5859b79ff4-lt9g6"] Nov 03 16:47:25.262144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.262107 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wr2\" (UniqueName: \"kubernetes.io/projected/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe-kube-api-access-k9wr2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.464063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.464026 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:47:25.471071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:25.471037 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-6d6854db88-ntxr5"] Nov 03 16:47:26.798490 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:26.798353 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c842719-571c-4cb7-a442-a88ad7ee53c7" path="/var/lib/kubelet/pods/3c842719-571c-4cb7-a442-a88ad7ee53c7/volumes" Nov 03 16:47:26.800114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:26.799656 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" path="/var/lib/kubelet/pods/f1a6628a-7e6e-46c3-b076-b98fb9dfdefe/volumes" Nov 03 16:47:26.909412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:26.909324 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" podUID="2655094a-621d-4a61-986d-257e16715040" containerName="payload-processing" containerID="cri-o://53c946c40437088e268e6c5f97a51ffc268528234da57bc710f04da661b344b0" gracePeriod=25 Nov 03 16:47:26.944293 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:26.944232 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" podUID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" containerName="payload-pre-processing" containerID="cri-o://cc3e8641ec3112a9e3b3829cc632d9e8b8ada436e5ab58d41515161ebd46e442" gracePeriod=25 Nov 03 16:47:27.138667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.137437 6001 generic.go:356] "Generic (PLEG): container finished" podID="2655094a-621d-4a61-986d-257e16715040" containerID="53c946c40437088e268e6c5f97a51ffc268528234da57bc710f04da661b344b0" exitCode=0 Nov 03 16:47:27.138667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.137581 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" event={"ID":"2655094a-621d-4a61-986d-257e16715040","Type":"ContainerDied","Data":"53c946c40437088e268e6c5f97a51ffc268528234da57bc710f04da661b344b0"} Nov 03 16:47:27.141492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.141017 6001 generic.go:356] "Generic (PLEG): container finished" podID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" containerID="cc3e8641ec3112a9e3b3829cc632d9e8b8ada436e5ab58d41515161ebd46e442" exitCode=0 Nov 03 16:47:27.141492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.141174 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" event={"ID":"c54964d5-88b6-498e-b9e2-8d0b92cd88e7","Type":"ContainerDied","Data":"cc3e8641ec3112a9e3b3829cc632d9e8b8ada436e5ab58d41515161ebd46e442"} Nov 03 16:47:27.233269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.233241 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:47:27.280099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.280065 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:47:27.398298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.398266 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\" (UniqueName: \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\") pod \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " Nov 03 16:47:27.398522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.398316 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\") pod \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\" (UID: \"c54964d5-88b6-498e-b9e2-8d0b92cd88e7\") " Nov 03 16:47:27.398522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.398357 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\" (UniqueName: \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\") pod \"2655094a-621d-4a61-986d-257e16715040\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " Nov 03 16:47:27.398696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.398673 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\") pod \"2655094a-621d-4a61-986d-257e16715040\" (UID: \"2655094a-621d-4a61-986d-257e16715040\") " Nov 03 16:47:27.398808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.398779 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume" pod "c54964d5-88b6-498e-b9e2-8d0b92cd88e7" (UID: "c54964d5-88b6-498e-b9e2-8d0b92cd88e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:27.399099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.399074 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume" pod "2655094a-621d-4a61-986d-257e16715040" (UID: "2655094a-621d-4a61-986d-257e16715040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:27.399402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.399382 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2655094a-621d-4a61-986d-257e16715040-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:27.399480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.399408 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:27.400720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.400696 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k" pod "c54964d5-88b6-498e-b9e2-8d0b92cd88e7" (UID: "c54964d5-88b6-498e-b9e2-8d0b92cd88e7"). InnerVolumeSpecName "kube-api-access-wcm4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:27.400889 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.400875 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst" pod "2655094a-621d-4a61-986d-257e16715040" (UID: "2655094a-621d-4a61-986d-257e16715040"). InnerVolumeSpecName "kube-api-access-77xst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:27.500888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.500853 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcm4k\" (UniqueName: \"kubernetes.io/projected/c54964d5-88b6-498e-b9e2-8d0b92cd88e7-kube-api-access-wcm4k\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:27.500888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:27.500886 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77xst\" (UniqueName: \"kubernetes.io/projected/2655094a-621d-4a61-986d-257e16715040-kube-api-access-77xst\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:28.150104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.150071 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" Nov 03 16:47:28.150104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.150098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m" event={"ID":"2655094a-621d-4a61-986d-257e16715040","Type":"ContainerDied","Data":"77f2371ae6f3962dc815e138150f4469e6547da56acf3420f8b3d0bd62142d51"} Nov 03 16:47:28.150668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.150168 6001 scope.go:122] "RemoveContainer" containerID="53c946c40437088e268e6c5f97a51ffc268528234da57bc710f04da661b344b0" Nov 03 16:47:28.151708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.151686 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" Nov 03 16:47:28.151803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.151720 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg" event={"ID":"c54964d5-88b6-498e-b9e2-8d0b92cd88e7","Type":"ContainerDied","Data":"0fcd03d9dbb81b19359a83c152c4f064965f211048795e3a0f41f1c494ecc01e"} Nov 03 16:47:28.162514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.162485 6001 scope.go:122] "RemoveContainer" containerID="cc3e8641ec3112a9e3b3829cc632d9e8b8ada436e5ab58d41515161ebd46e442" Nov 03 16:47:28.224898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.224861 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:47:28.232983 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.232955 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-6bdcdc9cb7-7ljzg"] Nov 03 16:47:28.256926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.256894 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:47:28.260160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.260126 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-78dd6dc855-k8p6m"] Nov 03 16:47:28.645289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.645206 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:47:28.646112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646087 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3c842719-571c-4cb7-a442-a88ad7ee53c7" containerName="payload-processing" Nov 03 16:47:28.646112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646109 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c842719-571c-4cb7-a442-a88ad7ee53c7" containerName="payload-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646127 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" containerName="payload-pre-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646132 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" containerName="payload-pre-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646147 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" containerName="payload-pre-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646152 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" containerName="payload-pre-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646170 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2655094a-621d-4a61-986d-257e16715040" containerName="payload-processing" Nov 03 16:47:28.646318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646175 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2655094a-621d-4a61-986d-257e16715040" containerName="payload-processing" Nov 03 16:47:28.646668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646351 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a6628a-7e6e-46c3-b076-b98fb9dfdefe" containerName="payload-pre-processing" Nov 03 16:47:28.646668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646368 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2655094a-621d-4a61-986d-257e16715040" containerName="payload-processing" Nov 03 16:47:28.646668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646383 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c842719-571c-4cb7-a442-a88ad7ee53c7" containerName="payload-processing" Nov 03 16:47:28.646668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.646391 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" containerName="payload-pre-processing" Nov 03 16:47:28.651603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.651580 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.671692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.671666 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b9e362-openshift-default-dockercfg-mrk2t\"" Nov 03 16:47:28.684947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.684918 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:47:28.737186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737151 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737187 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737323 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737423 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737489 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737524 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737589 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737661 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.737721 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.737699 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml4wk\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.791134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.791103 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2655094a-621d-4a61-986d-257e16715040" path="/var/lib/kubelet/pods/2655094a-621d-4a61-986d-257e16715040/volumes" Nov 03 16:47:28.791613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.791595 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c54964d5-88b6-498e-b9e2-8d0b92cd88e7" path="/var/lib/kubelet/pods/c54964d5-88b6-498e-b9e2-8d0b92cd88e7/volumes" Nov 03 16:47:28.838746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.838707 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.838949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.838826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.838949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.838871 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.838949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.838920 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.838966 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839030 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839069 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml4wk\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839107 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839135 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839176 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839432 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839569 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839476 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.839632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.839614 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.840109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.840087 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.841632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.841607 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.841728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.841641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.880366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.880204 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml4wk\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.890945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.890913 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\") pod \"e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:28.969018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:28.968920 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:29.313487 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:29.309427 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c42d69_d96d_4bf7_87ec_18dc52d1e2b7.slice/crio-7a26c12de89ea462b976e7a109574e8992f47b8b01d7e35d3f8ee77d6af947f5 WatchSource:0}: Error finding container 7a26c12de89ea462b976e7a109574e8992f47b8b01d7e35d3f8ee77d6af947f5: Status 404 returned error can't find the container with id 7a26c12de89ea462b976e7a109574e8992f47b8b01d7e35d3f8ee77d6af947f5 Nov 03 16:47:29.313487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.310576 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:47:29.313487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.313096 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:29.313487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.313176 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:29.313487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.313215 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:29.418006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.417961 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:47:29.422647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.422619 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.447596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.447563 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.447845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.447819 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.447924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.447876 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjfhg\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.447995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.447938 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.448057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.448013 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.448114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.448064 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.448114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.448103 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.448221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.448132 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.448221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.448187 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.455508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.455398 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d71e12-openshift-default-dockercfg-rdjxd\"" Nov 03 16:47:29.475669 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.475635 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:47:29.549628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549589 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.549815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549648 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjfhg\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.549815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.549815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549749 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.549815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549801 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549840 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549884 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549930 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.549968 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.550060 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.550251 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.550330 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.550495 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.550594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.550526 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.552103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.552077 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.552457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.552400 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.574538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.574476 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.574645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.574617 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjfhg\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\") pod \"e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:29.747456 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:29.747329 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:30.101778 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:30.101736 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cf8f64_79d8_4ec9_8ce5_b6616f7de71d.slice/crio-db8b5fdbb7eef0577101f2f97ae196234e0be1abf0d7349da37c22845e151620 WatchSource:0}: Error finding container db8b5fdbb7eef0577101f2f97ae196234e0be1abf0d7349da37c22845e151620: Status 404 returned error can't find the container with id db8b5fdbb7eef0577101f2f97ae196234e0be1abf0d7349da37c22845e151620 Nov 03 16:47:30.102130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.102107 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:47:30.105143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.105105 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:30.105242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.105187 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:30.105242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.105236 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:30.169322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.169280 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerStarted","Data":"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68"} Nov 03 16:47:30.169508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.169339 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerStarted","Data":"7a26c12de89ea462b976e7a109574e8992f47b8b01d7e35d3f8ee77d6af947f5"} Nov 03 16:47:30.170987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.170955 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerStarted","Data":"db8b5fdbb7eef0577101f2f97ae196234e0be1abf0d7349da37c22845e151620"} Nov 03 16:47:30.219714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.219645 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podStartSLOduration=2.219624285 podStartE2EDuration="2.219624285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:30.217570662 +0000 UTC m=+1945.550784231" watchObservedRunningTime="2026-11-03 16:47:30.219624285 +0000 UTC m=+1945.552837855" Nov 03 16:47:30.969133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.969098 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:30.974392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:30.974367 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:31.193266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.193223 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerStarted","Data":"c805ad85b7ec3902069a367edb74eb93b9a7e4dab069f2b5e45e76293576d2ef"} Nov 03 16:47:31.193494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.193342 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:31.200526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.200496 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:47:31.276423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.276364 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podStartSLOduration=2.276350454 podStartE2EDuration="2.276350454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:31.27412135 +0000 UTC m=+1946.607334920" watchObservedRunningTime="2026-11-03 16:47:31.276350454 +0000 UTC m=+1946.609564022" Nov 03 16:47:31.748254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.748222 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:31.754169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:31.754137 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:32.198410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:32.198321 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:32.199364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:32.199344 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:47:35.363161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.363118 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:47:35.367358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.367336 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.386373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.386340 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:47:35.395557 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.395531 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b9e362\"" Nov 03 16:47:35.420458 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.420422 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b9e362-dockercfg-s6gvq\"" Nov 03 16:47:35.444253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.444210 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.444464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.444429 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8k4\" (UniqueName: \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.450954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.450923 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:47:35.456428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.456403 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.479471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.474219 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:47:35.549918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.549853 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.550437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.550407 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8k4\" (UniqueName: \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.551107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.551086 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.551566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.551536 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5c9g\" (UniqueName: \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.552089 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.552069 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.576165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.576133 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8k4\" (UniqueName: \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\") pod \"payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.653250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.653151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5c9g\" (UniqueName: \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.653421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.653250 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.654017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.653989 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.678437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.678406 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5c9g\" (UniqueName: \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\") pod \"payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.682738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.682701 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:35.793136 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.792854 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:35.997605 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:35.997570 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc13cb10_885d_4150_90f6_0103e2564505.slice/crio-094e62a23cdc58ed489dd8eb2962c88aa31c236df96ae6e1279fcebcd2d51786 WatchSource:0}: Error finding container 094e62a23cdc58ed489dd8eb2962c88aa31c236df96ae6e1279fcebcd2d51786: Status 404 returned error can't find the container with id 094e62a23cdc58ed489dd8eb2962c88aa31c236df96ae6e1279fcebcd2d51786 Nov 03 16:47:35.998568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:35.998439 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:47:36.100523 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:36.100494 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1c8358_3904_49c1_b3cd_abf9a60b263d.slice/crio-f36b3615d733fd8518fe1e01f86f4293b6d080140c5a4de59c8c143596f46236 WatchSource:0}: Error finding container f36b3615d733fd8518fe1e01f86f4293b6d080140c5a4de59c8c143596f46236: Status 404 returned error can't find the container with id f36b3615d733fd8518fe1e01f86f4293b6d080140c5a4de59c8c143596f46236 Nov 03 16:47:36.101165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.101125 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:47:36.235229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.235193 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" event={"ID":"4f1c8358-3904-49c1-b3cd-abf9a60b263d","Type":"ContainerStarted","Data":"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586"} Nov 03 16:47:36.235426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.235244 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" event={"ID":"4f1c8358-3904-49c1-b3cd-abf9a60b263d","Type":"ContainerStarted","Data":"f36b3615d733fd8518fe1e01f86f4293b6d080140c5a4de59c8c143596f46236"} Nov 03 16:47:36.235426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.235412 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:36.237163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.237130 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" event={"ID":"cc13cb10-885d-4150-90f6-0103e2564505","Type":"ContainerStarted","Data":"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8"} Nov 03 16:47:36.237314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.237187 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" event={"ID":"cc13cb10-885d-4150-90f6-0103e2564505","Type":"ContainerStarted","Data":"094e62a23cdc58ed489dd8eb2962c88aa31c236df96ae6e1279fcebcd2d51786"} Nov 03 16:47:36.237314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.237216 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:36.282965 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.282844 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" podStartSLOduration=1.2828286119999999 podStartE2EDuration="1.282828612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:36.282581215 +0000 UTC m=+1951.615794785" watchObservedRunningTime="2026-11-03 16:47:36.282828612 +0000 UTC m=+1951.616042180" Nov 03 16:47:36.342681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:36.342613 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" podStartSLOduration=1.342593893 podStartE2EDuration="1.342593893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:36.337574326 +0000 UTC m=+1951.670787906" watchObservedRunningTime="2026-11-03 16:47:36.342593893 +0000 UTC m=+1951.675807466" Nov 03 16:47:37.349510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.349469 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:47:37.354401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.354379 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.372854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.372821 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d71e12\"" Nov 03 16:47:37.373643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.373618 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:47:37.382284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.382251 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d71e12-dockercfg-t27gc\"" Nov 03 16:47:37.443116 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.443074 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:47:37.448805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.448749 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.465952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.465916 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:47:37.492315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.492270 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvkl\" (UniqueName: \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.492653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.492621 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.609765 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.609859 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.609963 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxpb\" (UniqueName: \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.610032 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvkl\" (UniqueName: \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.610713 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.643151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.643118 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvkl\" (UniqueName: \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.671276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.670803 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:37.712068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.712028 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.712216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.712167 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxpb\" (UniqueName: \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.712992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.712965 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.752491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.752224 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxpb\" (UniqueName: \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\") pod \"payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.765523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.765487 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:37.977833 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:37.977797 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode037b66a_5d22_4550_a907_c9d7ad817ec8.slice/crio-9fc9860389eddc69d560753767d0a73ff42468edefe1f223ac5f9b006f72a52a WatchSource:0}: Error finding container 9fc9860389eddc69d560753767d0a73ff42468edefe1f223ac5f9b006f72a52a: Status 404 returned error can't find the container with id 9fc9860389eddc69d560753767d0a73ff42468edefe1f223ac5f9b006f72a52a Nov 03 16:47:37.979736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:37.979711 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:47:38.089664 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:38.089625 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c7db72_924b_4316_ac23_1caf7da1661c.slice/crio-2cc604a22305d68a28a5a4382cd5b0a3715c96e8f95cf84b7dfc61bd92d205d4 WatchSource:0}: Error finding container 2cc604a22305d68a28a5a4382cd5b0a3715c96e8f95cf84b7dfc61bd92d205d4: Status 404 returned error can't find the container with id 2cc604a22305d68a28a5a4382cd5b0a3715c96e8f95cf84b7dfc61bd92d205d4 Nov 03 16:47:38.089877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.089853 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:47:38.254877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.254772 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" event={"ID":"e037b66a-5d22-4550-a907-c9d7ad817ec8","Type":"ContainerStarted","Data":"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d"} Nov 03 16:47:38.254877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.254835 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" event={"ID":"e037b66a-5d22-4550-a907-c9d7ad817ec8","Type":"ContainerStarted","Data":"9fc9860389eddc69d560753767d0a73ff42468edefe1f223ac5f9b006f72a52a"} Nov 03 16:47:38.254877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.254863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:38.256723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.256689 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" event={"ID":"a2c7db72-924b-4316-ac23-1caf7da1661c","Type":"ContainerStarted","Data":"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3"} Nov 03 16:47:38.256898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.256753 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" event={"ID":"a2c7db72-924b-4316-ac23-1caf7da1661c","Type":"ContainerStarted","Data":"2cc604a22305d68a28a5a4382cd5b0a3715c96e8f95cf84b7dfc61bd92d205d4"} Nov 03 16:47:38.256898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.256781 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:38.300308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.300253 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" podStartSLOduration=1.300236574 podStartE2EDuration="1.300236574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:38.297093275 +0000 UTC m=+1953.630306840" watchObservedRunningTime="2026-11-03 16:47:38.300236574 +0000 UTC m=+1953.633450202" Nov 03 16:47:38.340154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:38.340092 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" podStartSLOduration=1.340076524 podStartE2EDuration="1.340076524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:38.339416136 +0000 UTC m=+1953.672629703" watchObservedRunningTime="2026-11-03 16:47:38.340076524 +0000 UTC m=+1953.673290092" Nov 03 16:47:42.970710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:42.970661 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:47:42.980237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:42.980213 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:42.991921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:42.991889 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:47:43.005313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.001952 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0a112\"" Nov 03 16:47:43.005313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.002723 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0a112-dockercfg-rhct8\"" Nov 03 16:47:43.102701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.102666 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72t7\" (UniqueName: \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.102866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.102737 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.283512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.283418 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72t7\" (UniqueName: \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.285882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.285848 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.288433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.287434 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.315542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.315507 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72t7\" (UniqueName: \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\") pod \"payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.615517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.615369 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:43.971826 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:43.971779 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df33f83_0636_43c8_b06e_576c8b2b93ae.slice/crio-a2e99ba27c08f99c11bfaace34a1ee9503dbb17ab2e5737937460781f7f95a19 WatchSource:0}: Error finding container a2e99ba27c08f99c11bfaace34a1ee9503dbb17ab2e5737937460781f7f95a19: Status 404 returned error can't find the container with id a2e99ba27c08f99c11bfaace34a1ee9503dbb17ab2e5737937460781f7f95a19 Nov 03 16:47:43.975299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:43.975274 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:47:44.171345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.171292 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:47:44.196548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.196512 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:47:44.334607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.334571 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" event={"ID":"1df33f83-0636-43c8-b06e-576c8b2b93ae","Type":"ContainerStarted","Data":"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e"} Nov 03 16:47:44.334815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.334632 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" event={"ID":"1df33f83-0636-43c8-b06e-576c8b2b93ae","Type":"ContainerStarted","Data":"a2e99ba27c08f99c11bfaace34a1ee9503dbb17ab2e5737937460781f7f95a19"} Nov 03 16:47:44.334815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.334661 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:44.400352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:44.400288 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" podStartSLOduration=2.400268121 podStartE2EDuration="2.400268121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:44.400098148 +0000 UTC m=+1959.733311718" watchObservedRunningTime="2026-11-03 16:47:44.400268121 +0000 UTC m=+1959.733481694" Nov 03 16:47:45.019778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:45.019734 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:47:45.080716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:45.080682 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:47:47.270785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:47.270752 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:47:47.271270 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:47.271048 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:47:49.172051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.171941 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" podUID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" containerName="payload-processing" containerID="cri-o://f7870f393e5bf0b3255d16f4e5c29f016f50b6fd02fbd2d30c19ae7b54a875ea" gracePeriod=25 Nov 03 16:47:49.197697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.197658 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" podUID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" containerName="payload-pre-processing" containerID="cri-o://2c5566085be9cb9a87739374a7220469ff2cc2a35dcfd41120754de43f1ce9d7" gracePeriod=25 Nov 03 16:47:49.269111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.269069 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:47:49.269423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.269401 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:47:49.424002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.423917 6001 generic.go:356] "Generic (PLEG): container finished" podID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" containerID="f7870f393e5bf0b3255d16f4e5c29f016f50b6fd02fbd2d30c19ae7b54a875ea" exitCode=0 Nov 03 16:47:49.424160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.424064 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" event={"ID":"5ba45132-fc0d-41c6-bf4c-c96823d6ff78","Type":"ContainerDied","Data":"f7870f393e5bf0b3255d16f4e5c29f016f50b6fd02fbd2d30c19ae7b54a875ea"} Nov 03 16:47:49.426080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.426049 6001 generic.go:356] "Generic (PLEG): container finished" podID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" containerID="2c5566085be9cb9a87739374a7220469ff2cc2a35dcfd41120754de43f1ce9d7" exitCode=0 Nov 03 16:47:49.426237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.426219 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" event={"ID":"64cf6a25-6b42-4bbb-9b02-764df6af86d5","Type":"ContainerDied","Data":"2c5566085be9cb9a87739374a7220469ff2cc2a35dcfd41120754de43f1ce9d7"} Nov 03 16:47:49.541157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.541117 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:47:49.559977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.559947 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:47:49.592727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.592691 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\" (UniqueName: \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\") pod \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " Nov 03 16:47:49.592926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.592854 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\" (UniqueName: \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\") pod \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " Nov 03 16:47:49.593012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.592997 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\") pod \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\" (UID: \"5ba45132-fc0d-41c6-bf4c-c96823d6ff78\") " Nov 03 16:47:49.593123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.593099 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\") pod \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\" (UID: \"64cf6a25-6b42-4bbb-9b02-764df6af86d5\") " Nov 03 16:47:49.593518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.593486 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume" pod "5ba45132-fc0d-41c6-bf4c-c96823d6ff78" (UID: "5ba45132-fc0d-41c6-bf4c-c96823d6ff78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:49.593890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.593863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume" pod "64cf6a25-6b42-4bbb-9b02-764df6af86d5" (UID: "64cf6a25-6b42-4bbb-9b02-764df6af86d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:49.594074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.594057 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:49.594139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.594080 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64cf6a25-6b42-4bbb-9b02-764df6af86d5-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:49.595329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.595309 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq" pod "64cf6a25-6b42-4bbb-9b02-764df6af86d5" (UID: "64cf6a25-6b42-4bbb-9b02-764df6af86d5"). InnerVolumeSpecName "kube-api-access-btxgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:49.595429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.595336 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl" pod "5ba45132-fc0d-41c6-bf4c-c96823d6ff78" (UID: "5ba45132-fc0d-41c6-bf4c-c96823d6ff78"). InnerVolumeSpecName "kube-api-access-7z7hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:49.695161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.695076 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btxgq\" (UniqueName: \"kubernetes.io/projected/64cf6a25-6b42-4bbb-9b02-764df6af86d5-kube-api-access-btxgq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:49.695161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:49.695110 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z7hl\" (UniqueName: \"kubernetes.io/projected/5ba45132-fc0d-41c6-bf4c-c96823d6ff78-kube-api-access-7z7hl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.021018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.020972 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" podUID="dbe8976e-5513-4196-abbf-81035edeac65" containerName="payload-processing" containerID="cri-o://ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367" gracePeriod=25 Nov 03 16:47:50.081762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.081686 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" podUID="75dc1547-aabb-43ee-91a4-72726183a03a" containerName="payload-pre-processing" containerID="cri-o://0e52f41961652c6ef71fed88522e93e5b5809bf33fdaee5b09774dc29356a39d" gracePeriod=25 Nov 03 16:47:50.425228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.424910 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:47:50.461036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.461000 6001 generic.go:356] "Generic (PLEG): container finished" podID="75dc1547-aabb-43ee-91a4-72726183a03a" containerID="0e52f41961652c6ef71fed88522e93e5b5809bf33fdaee5b09774dc29356a39d" exitCode=0 Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.461384 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" event={"ID":"75dc1547-aabb-43ee-91a4-72726183a03a","Type":"ContainerDied","Data":"0e52f41961652c6ef71fed88522e93e5b5809bf33fdaee5b09774dc29356a39d"} Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.467710 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbe8976e-5513-4196-abbf-81035edeac65" containerID="ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367" exitCode=0 Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.467798 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" event={"ID":"dbe8976e-5513-4196-abbf-81035edeac65","Type":"ContainerDied","Data":"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367"} Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.467840 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" event={"ID":"dbe8976e-5513-4196-abbf-81035edeac65","Type":"ContainerDied","Data":"7a462b9611005cb1180c7c3a65c685897481635d7faa48ff2c9a6f052165011a"} Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.467868 6001 scope.go:122] "RemoveContainer" containerID="ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367" Nov 03 16:47:50.475481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.468029 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7" Nov 03 16:47:50.476091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.475686 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" event={"ID":"5ba45132-fc0d-41c6-bf4c-c96823d6ff78","Type":"ContainerDied","Data":"4c5175e848a2096c5905fe22971a0070e315ff9644751179d194565a36d01b1f"} Nov 03 16:47:50.476091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.475856 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq" Nov 03 16:47:50.479780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.479750 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" event={"ID":"64cf6a25-6b42-4bbb-9b02-764df6af86d5","Type":"ContainerDied","Data":"ed608f547f015ac7b65cde236d834feb8eb9c33a4ff82f5734b37f77ad9eb8d4"} Nov 03 16:47:50.480151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.480035 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9" Nov 03 16:47:50.514194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.511825 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\") pod \"dbe8976e-5513-4196-abbf-81035edeac65\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " Nov 03 16:47:50.514194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.511953 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\" (UniqueName: \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\") pod \"dbe8976e-5513-4196-abbf-81035edeac65\" (UID: \"dbe8976e-5513-4196-abbf-81035edeac65\") " Nov 03 16:47:50.514194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.514150 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume" pod "dbe8976e-5513-4196-abbf-81035edeac65" (UID: "dbe8976e-5513-4196-abbf-81035edeac65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:50.542002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.541975 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:47:50.543258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.543224 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9" pod "dbe8976e-5513-4196-abbf-81035edeac65" (UID: "dbe8976e-5513-4196-abbf-81035edeac65"). InnerVolumeSpecName "kube-api-access-sfqv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:50.546958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.546939 6001 scope.go:122] "RemoveContainer" containerID="ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367" Nov 03 16:47:50.547484 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:50.547458 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367\": container with ID starting with ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367 not found: ID does not exist" containerID="ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367" Nov 03 16:47:50.547595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.547494 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367"} err="failed to get container status \"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367\": rpc error: code = NotFound desc = could not find container \"ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367\": container with ID starting with ea5a3dcbead4abd45728eb948011b203c914c416636bb1662ab7ea3b35e46367 not found: ID does not exist" Nov 03 16:47:50.547595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.547517 6001 scope.go:122] "RemoveContainer" containerID="f7870f393e5bf0b3255d16f4e5c29f016f50b6fd02fbd2d30c19ae7b54a875ea" Nov 03 16:47:50.561931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.561906 6001 scope.go:122] "RemoveContainer" containerID="2c5566085be9cb9a87739374a7220469ff2cc2a35dcfd41120754de43f1ce9d7" Nov 03 16:47:50.574562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.574518 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:47:50.575435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.575407 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-98e3b9-84bb5845f5-f8qk9"] Nov 03 16:47:50.622493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.622365 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\" (UniqueName: \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\") pod \"75dc1547-aabb-43ee-91a4-72726183a03a\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " Nov 03 16:47:50.622675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.622589 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\") pod \"75dc1547-aabb-43ee-91a4-72726183a03a\" (UID: \"75dc1547-aabb-43ee-91a4-72726183a03a\") " Nov 03 16:47:50.623348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.623316 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe8976e-5513-4196-abbf-81035edeac65-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.623348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.623349 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqv9\" (UniqueName: \"kubernetes.io/projected/dbe8976e-5513-4196-abbf-81035edeac65-kube-api-access-sfqv9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.624127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.624098 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume" pod "75dc1547-aabb-43ee-91a4-72726183a03a" (UID: "75dc1547-aabb-43ee-91a4-72726183a03a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:50.656466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.633226 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc" pod "75dc1547-aabb-43ee-91a4-72726183a03a" (UID: "75dc1547-aabb-43ee-91a4-72726183a03a"). InnerVolumeSpecName "kube-api-access-jfwrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:50.667996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.667951 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:47:50.694472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.689532 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-98e3b9-bd69b48f5-ws6jq"] Nov 03 16:47:50.738473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.734872 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfwrc\" (UniqueName: \"kubernetes.io/projected/75dc1547-aabb-43ee-91a4-72726183a03a-kube-api-access-jfwrc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.738473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.734909 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75dc1547-aabb-43ee-91a4-72726183a03a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:47:50.839352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.839324 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" path="/var/lib/kubelet/pods/5ba45132-fc0d-41c6-bf4c-c96823d6ff78/volumes" Nov 03 16:47:50.840002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.839976 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" path="/var/lib/kubelet/pods/64cf6a25-6b42-4bbb-9b02-764df6af86d5/volumes" Nov 03 16:47:50.868396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.868357 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:47:50.875098 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:50.875018 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-b6cc69c-ltjg7"] Nov 03 16:47:51.489568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.489537 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" Nov 03 16:47:51.490036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.489568 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd" event={"ID":"75dc1547-aabb-43ee-91a4-72726183a03a","Type":"ContainerDied","Data":"d5fdde38fe8362ec4105ddf1e48a78781668bd25626199da11619773f2762c0c"} Nov 03 16:47:51.490036 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.489630 6001 scope.go:122] "RemoveContainer" containerID="0e52f41961652c6ef71fed88522e93e5b5809bf33fdaee5b09774dc29356a39d" Nov 03 16:47:51.546319 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.546288 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:47:51.558217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.558175 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-56df4b5dff-xzptd"] Nov 03 16:47:51.869132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869028 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:47:51.869739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869717 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbe8976e-5513-4196-abbf-81035edeac65" containerName="payload-processing" Nov 03 16:47:51.869739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869733 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe8976e-5513-4196-abbf-81035edeac65" containerName="payload-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869751 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" containerName="payload-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869757 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" containerName="payload-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869770 6001 cpu_manager.go:414] "containerMap: removing container" podUID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" containerName="payload-pre-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869776 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" containerName="payload-pre-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869816 6001 cpu_manager.go:414] "containerMap: removing container" podUID="75dc1547-aabb-43ee-91a4-72726183a03a" containerName="payload-pre-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869821 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75dc1547-aabb-43ee-91a4-72726183a03a" containerName="payload-pre-processing" Nov 03 16:47:51.869918 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869922 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75dc1547-aabb-43ee-91a4-72726183a03a" containerName="payload-pre-processing" Nov 03 16:47:51.870285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869933 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe8976e-5513-4196-abbf-81035edeac65" containerName="payload-processing" Nov 03 16:47:51.870285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869944 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba45132-fc0d-41c6-bf4c-c96823d6ff78" containerName="payload-processing" Nov 03 16:47:51.870285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.869952 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cf6a25-6b42-4bbb-9b02-764df6af86d5" containerName="payload-pre-processing" Nov 03 16:47:51.873398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.873372 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:51.890331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.890300 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad7bf6-dockercfg-8sprn\"" Nov 03 16:47:51.891974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.891948 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad7bf6\"" Nov 03 16:47:51.902801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.902776 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:47:51.954613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.954581 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:51.954799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.954750 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74zf\" (UniqueName: \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:51.962153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.962128 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:47:51.965345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.965320 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:51.985302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:51.985267 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:47:52.056707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.056660 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.056949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.056915 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:52.057088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.057021 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfwj\" (UniqueName: \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.057161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.057144 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74zf\" (UniqueName: \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:52.057753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.057728 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:52.084405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.084360 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74zf\" (UniqueName: \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:52.167583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.158998 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.167583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.159239 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfwj\" (UniqueName: \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.167583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.160469 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.191467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.191363 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:52.196463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.193706 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfwj\" (UniqueName: \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\") pod \"payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.279558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.279506 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:52.302884 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:47:52.302691 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:47:52.577628 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:52.577576 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab8bcb5_bd34_4797_a266_32cda101e682.slice/crio-985abd572007726adca967578787f1a20e2f03df53014452fe116d42f179084d WatchSource:0}: Error finding container 985abd572007726adca967578787f1a20e2f03df53014452fe116d42f179084d: Status 404 returned error can't find the container with id 985abd572007726adca967578787f1a20e2f03df53014452fe116d42f179084d Nov 03 16:47:52.584015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.583987 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:47:52.669194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.669164 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:47:52.672784 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:47:52.672757 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ea57ba_1706_493f_bd7b_f90a5117b655.slice/crio-ce07b238552522aa227305ceb4f07f031811284f2567e646d41fea556bf381d7 WatchSource:0}: Error finding container ce07b238552522aa227305ceb4f07f031811284f2567e646d41fea556bf381d7: Status 404 returned error can't find the container with id ce07b238552522aa227305ceb4f07f031811284f2567e646d41fea556bf381d7 Nov 03 16:47:52.794043 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.794007 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75dc1547-aabb-43ee-91a4-72726183a03a" path="/var/lib/kubelet/pods/75dc1547-aabb-43ee-91a4-72726183a03a/volumes" Nov 03 16:47:52.794670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:52.794646 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe8976e-5513-4196-abbf-81035edeac65" path="/var/lib/kubelet/pods/dbe8976e-5513-4196-abbf-81035edeac65/volumes" Nov 03 16:47:53.514064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.514026 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" event={"ID":"b4ea57ba-1706-493f-bd7b-f90a5117b655","Type":"ContainerStarted","Data":"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031"} Nov 03 16:47:53.514308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.514124 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" event={"ID":"b4ea57ba-1706-493f-bd7b-f90a5117b655","Type":"ContainerStarted","Data":"ce07b238552522aa227305ceb4f07f031811284f2567e646d41fea556bf381d7"} Nov 03 16:47:53.514308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.514256 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:47:53.515768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.515747 6001 generic.go:356] "Generic (PLEG): container finished" podID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerID="ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b" exitCode=0 Nov 03 16:47:53.515899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.515815 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerDied","Data":"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b"} Nov 03 16:47:53.516231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.516212 6001 scope.go:122] "RemoveContainer" containerID="ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b" Nov 03 16:47:53.517666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517644 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" event={"ID":"cab8bcb5-bd34-4797-a266-32cda101e682","Type":"ContainerStarted","Data":"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742"} Nov 03 16:47:53.517755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517689 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" event={"ID":"cab8bcb5-bd34-4797-a266-32cda101e682","Type":"ContainerStarted","Data":"985abd572007726adca967578787f1a20e2f03df53014452fe116d42f179084d"} Nov 03 16:47:53.517755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517712 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:47:53.517905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517878 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:53.517958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517945 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:53.517993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.517980 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:47:53.558096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.558006 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" podStartSLOduration=2.557993745 podStartE2EDuration="2.557993745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:53.557424353 +0000 UTC m=+1968.890637930" watchObservedRunningTime="2026-11-03 16:47:53.557993745 +0000 UTC m=+1968.891207334" Nov 03 16:47:53.663491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:53.662530 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" podStartSLOduration=2.662510218 podStartE2EDuration="2.662510218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:53.656183153 +0000 UTC m=+1968.989396726" watchObservedRunningTime="2026-11-03 16:47:53.662510218 +0000 UTC m=+1968.995723790" Nov 03 16:47:54.527211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:54.527171 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerStarted","Data":"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b"} Nov 03 16:47:54.819542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:54.819466 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:47:54.819914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:54.819641 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 16:47:54.819914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:54.819687 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 16:47:55.366469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:55.364600 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:47:55.820261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:55.820223 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 16:47:55.820729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:55.820282 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 16:47:56.820675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:56.820639 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 16:47:56.821085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:56.820691 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 16:47:57.819521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:57.819481 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 16:47:57.819725 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:57.819541 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 16:47:58.820357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:58.820316 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Nov 03 16:47:58.820805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:58.820373 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Nov 03 16:47:59.871216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:59.871176 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:47:59.872241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:59.872187 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:47:59.873820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:59.873791 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:47:59.972403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:47:59.972363 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:48:00.036653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:00.035412 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:48:04.531229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:04.531199 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:48:04.531712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:04.531269 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:48:05.036588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.036535 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" podUID="5411db87-17cc-4d9e-ac24-9f9fbda98056" containerName="payload-pre-processing" containerID="cri-o://f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230" gracePeriod=25 Nov 03 16:48:05.317537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.317505 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:48:05.399123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.399073 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\" (UniqueName: \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\") pod \"5411db87-17cc-4d9e-ac24-9f9fbda98056\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " Nov 03 16:48:05.399354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.399208 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\") pod \"5411db87-17cc-4d9e-ac24-9f9fbda98056\" (UID: \"5411db87-17cc-4d9e-ac24-9f9fbda98056\") " Nov 03 16:48:05.399622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.399586 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume" pod "5411db87-17cc-4d9e-ac24-9f9fbda98056" (UID: "5411db87-17cc-4d9e-ac24-9f9fbda98056"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:05.400082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.400060 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5411db87-17cc-4d9e-ac24-9f9fbda98056-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:05.401623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.401583 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp" pod "5411db87-17cc-4d9e-ac24-9f9fbda98056" (UID: "5411db87-17cc-4d9e-ac24-9f9fbda98056"). InnerVolumeSpecName "kube-api-access-zqrqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:05.501349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.501308 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqrqp\" (UniqueName: \"kubernetes.io/projected/5411db87-17cc-4d9e-ac24-9f9fbda98056-kube-api-access-zqrqp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:05.602957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.602853 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" podUID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" containerName="payload-processing" containerID="cri-o://466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4" gracePeriod=25 Nov 03 16:48:05.643943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.643904 6001 generic.go:356] "Generic (PLEG): container finished" podID="5411db87-17cc-4d9e-ac24-9f9fbda98056" containerID="f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230" exitCode=0 Nov 03 16:48:05.644120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.643957 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" Nov 03 16:48:05.644120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.643982 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" event={"ID":"5411db87-17cc-4d9e-ac24-9f9fbda98056","Type":"ContainerDied","Data":"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230"} Nov 03 16:48:05.644120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.644053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn" event={"ID":"5411db87-17cc-4d9e-ac24-9f9fbda98056","Type":"ContainerDied","Data":"cec33d36f39ed456f5073ca3b9ed5a546254b03991f4b1d11cbda5225d18bf84"} Nov 03 16:48:05.644120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.644085 6001 scope.go:122] "RemoveContainer" containerID="f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230" Nov 03 16:48:05.704869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.704837 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:48:05.714779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.714738 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-87b132-8574cff4ff-hqgfn"] Nov 03 16:48:05.718111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.718087 6001 scope.go:122] "RemoveContainer" containerID="f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230" Nov 03 16:48:05.718717 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:05.718690 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230\": container with ID starting with f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230 not found: ID does not exist" containerID="f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230" Nov 03 16:48:05.718822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.718729 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230"} err="failed to get container status \"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230\": rpc error: code = NotFound desc = could not find container \"f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230\": container with ID starting with f896cd748c0f49b6dbdc16763b5afa6efbbdfed45af46a3c8d0e9c312583e230 not found: ID does not exist" Nov 03 16:48:05.979648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.978906 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:48:05.980114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.980087 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5411db87-17cc-4d9e-ac24-9f9fbda98056" containerName="payload-pre-processing" Nov 03 16:48:05.980201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.980115 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5411db87-17cc-4d9e-ac24-9f9fbda98056" containerName="payload-pre-processing" Nov 03 16:48:05.980351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.980333 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5411db87-17cc-4d9e-ac24-9f9fbda98056" containerName="payload-pre-processing" Nov 03 16:48:05.988011 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.987981 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:05.995778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:05.995748 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:48:06.032511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.020769 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-9947ab-dockercfg-8rvm8\"" Nov 03 16:48:06.032511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.021541 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-9947ab\"" Nov 03 16:48:06.035088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.035037 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:48:06.103481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.103432 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:48:06.104860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.104827 6001 cpu_manager.go:414] "containerMap: removing container" podUID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" containerName="payload-processing" Nov 03 16:48:06.105077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.105062 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" containerName="payload-processing" Nov 03 16:48:06.105387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.105374 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" containerName="payload-processing" Nov 03 16:48:06.112278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.112247 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.131754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.131719 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\") pod \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " Nov 03 16:48:06.133724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.133695 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\" (UniqueName: \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\") pod \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\" (UID: \"025b982b-9cee-4ce1-84fe-7d5a04632f2d\") " Nov 03 16:48:06.134460 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.134414 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r946p\" (UniqueName: \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.135403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.135379 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.136285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.135712 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume" pod "025b982b-9cee-4ce1-84fe-7d5a04632f2d" (UID: "025b982b-9cee-4ce1-84fe-7d5a04632f2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:06.139040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.138992 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:48:06.142997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.142949 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf" pod "025b982b-9cee-4ce1-84fe-7d5a04632f2d" (UID: "025b982b-9cee-4ce1-84fe-7d5a04632f2d"). InnerVolumeSpecName "kube-api-access-rgpcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:06.237508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237417 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r946p\" (UniqueName: \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.237683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237581 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.237683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237637 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.237810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237705 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prm57\" (UniqueName: \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.237947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237931 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025b982b-9cee-4ce1-84fe-7d5a04632f2d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:06.237990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.237949 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgpcf\" (UniqueName: \"kubernetes.io/projected/025b982b-9cee-4ce1-84fe-7d5a04632f2d-kube-api-access-rgpcf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:06.238292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.238273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.261981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.261946 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r946p\" (UniqueName: \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\") pod \"payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.337942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.337904 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:06.338831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.338800 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.338910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.338876 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prm57\" (UniqueName: \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.339530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.339506 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.363421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.363385 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prm57\" (UniqueName: \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\") pod \"payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.452971 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.446518 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:06.645170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.645132 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:48:06.645694 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:48:06.645666 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e393d06_5b20_44fd_b24e_efa3cc0f0665.slice/crio-447fced1e1e1cfeda65bd516c9d2ad18d7cbb4f056e9390eb16e59525a499a41 WatchSource:0}: Error finding container 447fced1e1e1cfeda65bd516c9d2ad18d7cbb4f056e9390eb16e59525a499a41: Status 404 returned error can't find the container with id 447fced1e1e1cfeda65bd516c9d2ad18d7cbb4f056e9390eb16e59525a499a41 Nov 03 16:48:06.655399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.655372 6001 generic.go:356] "Generic (PLEG): container finished" podID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" containerID="466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4" exitCode=0 Nov 03 16:48:06.655545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.655436 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" Nov 03 16:48:06.655545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.655465 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" event={"ID":"025b982b-9cee-4ce1-84fe-7d5a04632f2d","Type":"ContainerDied","Data":"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4"} Nov 03 16:48:06.655545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.655527 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5" event={"ID":"025b982b-9cee-4ce1-84fe-7d5a04632f2d","Type":"ContainerDied","Data":"6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8"} Nov 03 16:48:06.655702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.655557 6001 scope.go:122] "RemoveContainer" containerID="466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4" Nov 03 16:48:06.669256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.669235 6001 scope.go:122] "RemoveContainer" containerID="466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4" Nov 03 16:48:06.669585 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:06.669559 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4\": container with ID starting with 466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4 not found: ID does not exist" containerID="466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4" Nov 03 16:48:06.669677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.669596 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4"} err="failed to get container status \"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4\": rpc error: code = NotFound desc = could not find container \"466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4\": container with ID starting with 466d18f5614192196513e15c1aca3773d595a0737964c08422d98554f7172eb4 not found: ID does not exist" Nov 03 16:48:06.728698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.728658 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:48:06.737564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.737536 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-87b132-57c4556bc9-xnvh5"] Nov 03 16:48:06.751708 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:06.751668 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025b982b_9cee_4ce1_84fe_7d5a04632f2d.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025b982b_9cee_4ce1_84fe_7d5a04632f2d.slice/crio-6c9f39f814d0c68bc03f5cbc0de6ebeca6fa978684cc7feb8b7fc258ad1427e8\": RecentStats: unable to find data in memory cache]" Nov 03 16:48:06.799724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.799692 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025b982b-9cee-4ce1-84fe-7d5a04632f2d" path="/var/lib/kubelet/pods/025b982b-9cee-4ce1-84fe-7d5a04632f2d/volumes" Nov 03 16:48:06.800216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.800193 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5411db87-17cc-4d9e-ac24-9f9fbda98056" path="/var/lib/kubelet/pods/5411db87-17cc-4d9e-ac24-9f9fbda98056/volumes" Nov 03 16:48:06.800610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:06.800587 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:48:07.668029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.667977 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" event={"ID":"6e393d06-5b20-44fd-b24e-efa3cc0f0665","Type":"ContainerStarted","Data":"186064aee239c3bc2a1ca4328bb0b717aa703d21da800c11e4b50c1e67567110"} Nov 03 16:48:07.668562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.668055 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:07.668562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.668079 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" event={"ID":"6e393d06-5b20-44fd-b24e-efa3cc0f0665","Type":"ContainerStarted","Data":"447fced1e1e1cfeda65bd516c9d2ad18d7cbb4f056e9390eb16e59525a499a41"} Nov 03 16:48:07.671219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.671185 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" event={"ID":"1321a470-df83-4378-ad62-fbb74b56a207","Type":"ContainerStarted","Data":"5fbe8fa871cd668a9bf51eb08b2a6fbd2bc65021a4ab03b57ad36c9ebae82b59"} Nov 03 16:48:07.671338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.671235 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" event={"ID":"1321a470-df83-4378-ad62-fbb74b56a207","Type":"ContainerStarted","Data":"1c2e1ca9dc80d1a3eeabb9363f46ad284fa70f30017fc2573eaeb106d49cc9c0"} Nov 03 16:48:07.671338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.671278 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:07.712640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.712588 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" podStartSLOduration=2.712573632 podStartE2EDuration="2.712573632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:07.711674734 +0000 UTC m=+1983.044888305" watchObservedRunningTime="2026-11-03 16:48:07.712573632 +0000 UTC m=+1983.045787197" Nov 03 16:48:07.757554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:07.757484 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" podStartSLOduration=1.757464545 podStartE2EDuration="1.757464545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:07.75539707 +0000 UTC m=+1983.088610641" watchObservedRunningTime="2026-11-03 16:48:07.757464545 +0000 UTC m=+1983.090678115" Nov 03 16:48:11.031128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:11.031091 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn"] Nov 03 16:48:11.055477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:11.043966 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee5c5d4-dhpdn"] Nov 03 16:48:12.792305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:12.792265 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a697ecc-347b-4296-a3d1-c36d9d4c68ee" path="/var/lib/kubelet/pods/2a697ecc-347b-4296-a3d1-c36d9d4c68ee/volumes" Nov 03 16:48:17.233687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:17.233593 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:48:17.233687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:17.233675 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:48:18.029400 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:18.029358 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd"] Nov 03 16:48:18.041627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:18.041583 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5608dbd3-g6rdd"] Nov 03 16:48:18.681936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:18.681903 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:48:18.682535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:18.681979 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:48:18.799193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:18.799146 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b0311d3-08a8-46e9-a80e-182ab828be9c" path="/var/lib/kubelet/pods/3b0311d3-08a8-46e9-a80e-182ab828be9c/volumes" Nov 03 16:48:20.815425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.815390 6001 generic.go:356] "Generic (PLEG): container finished" podID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerID="58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68" exitCode=0 Nov 03 16:48:20.815954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.815493 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerDied","Data":"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68"} Nov 03 16:48:20.816007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.815979 6001 scope.go:122] "RemoveContainer" containerID="58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68" Nov 03 16:48:20.817646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.817611 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:20.817776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.817706 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:20.817776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:20.817758 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:21.825015 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:21.824976 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerStarted","Data":"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c"} Nov 03 16:48:21.969258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:21.969222 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:48:21.969545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:21.969525 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:21.969603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:21.969574 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:22.277735 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:22.277698 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:48:22.970074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:22.970025 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:22.970683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:22.970093 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:23.970095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:23.970048 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:23.970662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:23.970124 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:24.970056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:24.970008 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:24.970247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:24.970101 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:25.970038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:25.969994 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:25.970246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:25.970089 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:26.970486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:26.969517 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Nov 03 16:48:26.970486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:26.969595 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Nov 03 16:48:27.975493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:27.975435 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:48:27.975935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:27.975875 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:48:27.976671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:27.976651 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:48:29.992096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:29.992057 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:48:30.051617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:30.051579 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:48:32.033258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:32.033210 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x"] Nov 03 16:48:32.037826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:32.037775 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f64772-z2n5x"] Nov 03 16:48:32.795467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:32.794940 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9c50c16-3bea-4747-9f91-e77f71e18fa2" path="/var/lib/kubelet/pods/e9c50c16-3bea-4747-9f91-e77f71e18fa2/volumes" Nov 03 16:48:33.972909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:33.972870 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:48:33.973438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:33.973143 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" podUID="1df33f83-0636-43c8-b06e-576c8b2b93ae" containerName="payload-processing" containerID="cri-o://1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e" gracePeriod=30 Nov 03 16:48:34.077591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.077553 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:48:34.082226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.082199 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.093674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.093552 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:48:34.103565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.102358 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a7d95\"" Nov 03 16:48:34.103565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.102484 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a7d95-dockercfg-km724\"" Nov 03 16:48:34.122322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.107756 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:48:34.122322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.113739 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.134950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.134915 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:48:34.193389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.193349 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mskg2\" (UniqueName: \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.193705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.193672 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.193795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.193773 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4j4\" (UniqueName: \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.193959 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.193942 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.295812 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.295772 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4j4\" (UniqueName: \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.296004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.295936 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.296004 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.295982 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mskg2\" (UniqueName: \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.296124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.296087 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.296922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.296893 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.297072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.297049 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.320957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.320917 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4j4\" (UniqueName: \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\") pod \"payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.321703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.321682 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mskg2\" (UniqueName: \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.430519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.416834 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:34.453469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.440769 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:34.489556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.483068 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:48:34.634411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.634372 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\") pod \"1df33f83-0636-43c8-b06e-576c8b2b93ae\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " Nov 03 16:48:34.634860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.634485 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\" (UniqueName: \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\") pod \"1df33f83-0636-43c8-b06e-576c8b2b93ae\" (UID: \"1df33f83-0636-43c8-b06e-576c8b2b93ae\") " Nov 03 16:48:34.638845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.635133 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume" pod "1df33f83-0636-43c8-b06e-576c8b2b93ae" (UID: "1df33f83-0636-43c8-b06e-576c8b2b93ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:34.638845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.637599 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7" pod "1df33f83-0636-43c8-b06e-576c8b2b93ae" (UID: "1df33f83-0636-43c8-b06e-576c8b2b93ae"). InnerVolumeSpecName "kube-api-access-q72t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:34.736410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.736372 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df33f83-0636-43c8-b06e-576c8b2b93ae-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.736410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.736402 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q72t7\" (UniqueName: \"kubernetes.io/projected/1df33f83-0636-43c8-b06e-576c8b2b93ae-kube-api-access-q72t7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.896326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.896198 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:48:34.898203 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:48:34.897867 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22bd8306_8599_45f6_994a_9d420b207179.slice/crio-ff730a805421f509b89c070112ceba0602532ef154e6aeda279cfa7a4a8b1eff WatchSource:0}: Error finding container ff730a805421f509b89c070112ceba0602532ef154e6aeda279cfa7a4a8b1eff: Status 404 returned error can't find the container with id ff730a805421f509b89c070112ceba0602532ef154e6aeda279cfa7a4a8b1eff Nov 03 16:48:34.900690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.900663 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:48:34.992586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.992414 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" podUID="a2c7db72-924b-4316-ac23-1caf7da1661c" containerName="payload-processing" containerID="cri-o://d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3" gracePeriod=25 Nov 03 16:48:34.993588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.993564 6001 generic.go:356] "Generic (PLEG): container finished" podID="1df33f83-0636-43c8-b06e-576c8b2b93ae" containerID="1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e" exitCode=0 Nov 03 16:48:34.993683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.993645 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" Nov 03 16:48:34.993683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.993649 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" event={"ID":"1df33f83-0636-43c8-b06e-576c8b2b93ae","Type":"ContainerDied","Data":"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e"} Nov 03 16:48:34.993803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.993736 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp" event={"ID":"1df33f83-0636-43c8-b06e-576c8b2b93ae","Type":"ContainerDied","Data":"a2e99ba27c08f99c11bfaace34a1ee9503dbb17ab2e5737937460781f7f95a19"} Nov 03 16:48:34.993803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.993771 6001 scope.go:122] "RemoveContainer" containerID="1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e" Nov 03 16:48:34.995341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.995316 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" event={"ID":"d98ef26e-4b33-45e5-af89-36e2f29af9d2","Type":"ContainerStarted","Data":"f20d4fdae754fdc06e8bdf2147854db918e743a6a7e7e855052695e94aa00778"} Nov 03 16:48:34.998588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:34.998565 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" event={"ID":"22bd8306-8599-45f6-994a-9d420b207179","Type":"ContainerStarted","Data":"ff730a805421f509b89c070112ceba0602532ef154e6aeda279cfa7a4a8b1eff"} Nov 03 16:48:35.068471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.064524 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" podUID="e037b66a-5d22-4550-a907-c9d7ad817ec8" containerName="payload-pre-processing" containerID="cri-o://4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d" gracePeriod=25 Nov 03 16:48:35.076640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.076612 6001 scope.go:122] "RemoveContainer" containerID="1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e" Nov 03 16:48:35.077096 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:35.077065 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e\": container with ID starting with 1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e not found: ID does not exist" containerID="1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e" Nov 03 16:48:35.077216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.077104 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e"} err="failed to get container status \"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e\": rpc error: code = NotFound desc = could not find container \"1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e\": container with ID starting with 1ed147623e0c2c004f3a77af2f8f5603723fce4fe5e90ec4664e17f1895b0b4e not found: ID does not exist" Nov 03 16:48:35.111509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.111468 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:48:35.117865 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.117827 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e0a112-6d86c7c7bd-w6grp"] Nov 03 16:48:35.405065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.405037 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:48:35.473505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.449655 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:48:35.478662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.478631 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\") pod \"a2c7db72-924b-4316-ac23-1caf7da1661c\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " Nov 03 16:48:35.478846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.478733 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\" (UniqueName: \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\") pod \"a2c7db72-924b-4316-ac23-1caf7da1661c\" (UID: \"a2c7db72-924b-4316-ac23-1caf7da1661c\") " Nov 03 16:48:35.479866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.479823 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume" pod "a2c7db72-924b-4316-ac23-1caf7da1661c" (UID: "a2c7db72-924b-4316-ac23-1caf7da1661c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:35.485624 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.485577 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb" pod "a2c7db72-924b-4316-ac23-1caf7da1661c" (UID: "a2c7db72-924b-4316-ac23-1caf7da1661c"). InnerVolumeSpecName "kube-api-access-7gxpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:35.580165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.580127 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\") pod \"e037b66a-5d22-4550-a907-c9d7ad817ec8\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " Nov 03 16:48:35.580517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.580492 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\" (UniqueName: \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\") pod \"e037b66a-5d22-4550-a907-c9d7ad817ec8\" (UID: \"e037b66a-5d22-4550-a907-c9d7ad817ec8\") " Nov 03 16:48:35.580646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.580545 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume" pod "e037b66a-5d22-4550-a907-c9d7ad817ec8" (UID: "e037b66a-5d22-4550-a907-c9d7ad817ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:35.581069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.581047 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c7db72-924b-4316-ac23-1caf7da1661c-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:35.581069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.581069 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxpb\" (UniqueName: \"kubernetes.io/projected/a2c7db72-924b-4316-ac23-1caf7da1661c-kube-api-access-7gxpb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:35.581180 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.581079 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e037b66a-5d22-4550-a907-c9d7ad817ec8-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:35.582652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.582626 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl" pod "e037b66a-5d22-4550-a907-c9d7ad817ec8" (UID: "e037b66a-5d22-4550-a907-c9d7ad817ec8"). InnerVolumeSpecName "kube-api-access-9hvkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:35.683117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:35.683027 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hvkl\" (UniqueName: \"kubernetes.io/projected/e037b66a-5d22-4550-a907-c9d7ad817ec8-kube-api-access-9hvkl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:36.008250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.008210 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" event={"ID":"22bd8306-8599-45f6-994a-9d420b207179","Type":"ContainerStarted","Data":"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044"} Nov 03 16:48:36.008701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.008472 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:36.009858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.009830 6001 generic.go:356] "Generic (PLEG): container finished" podID="a2c7db72-924b-4316-ac23-1caf7da1661c" containerID="d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3" exitCode=0 Nov 03 16:48:36.010002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.009869 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" Nov 03 16:48:36.010002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.009907 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" event={"ID":"a2c7db72-924b-4316-ac23-1caf7da1661c","Type":"ContainerDied","Data":"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3"} Nov 03 16:48:36.010002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.009956 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl" event={"ID":"a2c7db72-924b-4316-ac23-1caf7da1661c","Type":"ContainerDied","Data":"2cc604a22305d68a28a5a4382cd5b0a3715c96e8f95cf84b7dfc61bd92d205d4"} Nov 03 16:48:36.010002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.009980 6001 scope.go:122] "RemoveContainer" containerID="d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3" Nov 03 16:48:36.011779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.011755 6001 generic.go:356] "Generic (PLEG): container finished" podID="e037b66a-5d22-4550-a907-c9d7ad817ec8" containerID="4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d" exitCode=0 Nov 03 16:48:36.011916 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.011801 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" Nov 03 16:48:36.011916 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.011837 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" event={"ID":"e037b66a-5d22-4550-a907-c9d7ad817ec8","Type":"ContainerDied","Data":"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d"} Nov 03 16:48:36.011916 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.011878 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66" event={"ID":"e037b66a-5d22-4550-a907-c9d7ad817ec8","Type":"ContainerDied","Data":"9fc9860389eddc69d560753767d0a73ff42468edefe1f223ac5f9b006f72a52a"} Nov 03 16:48:36.014902 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.014872 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" event={"ID":"d98ef26e-4b33-45e5-af89-36e2f29af9d2","Type":"ContainerStarted","Data":"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99"} Nov 03 16:48:36.015006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.014954 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:36.021933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.021907 6001 scope.go:122] "RemoveContainer" containerID="d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3" Nov 03 16:48:36.022255 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:36.022226 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3\": container with ID starting with d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3 not found: ID does not exist" containerID="d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3" Nov 03 16:48:36.022353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.022264 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3"} err="failed to get container status \"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3\": rpc error: code = NotFound desc = could not find container \"d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3\": container with ID starting with d8f9aced53109a46c6f7aaef1e7dd0792672a9e35610204539dec41e28a3e7e3 not found: ID does not exist" Nov 03 16:48:36.022353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.022287 6001 scope.go:122] "RemoveContainer" containerID="4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d" Nov 03 16:48:36.033390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.033368 6001 scope.go:122] "RemoveContainer" containerID="4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d" Nov 03 16:48:36.033729 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:36.033708 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d\": container with ID starting with 4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d not found: ID does not exist" containerID="4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d" Nov 03 16:48:36.033780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.033739 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d"} err="failed to get container status \"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d\": rpc error: code = NotFound desc = could not find container \"4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d\": container with ID starting with 4bf3906b6968165df8cc776a1d47c3c9e2eccd2cff8383462695148a0bcc424d not found: ID does not exist" Nov 03 16:48:36.072910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.072840 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" podStartSLOduration=2.072825075 podStartE2EDuration="2.072825075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:36.068804081 +0000 UTC m=+2011.402017682" watchObservedRunningTime="2026-11-03 16:48:36.072825075 +0000 UTC m=+2011.406038694" Nov 03 16:48:36.109191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.109157 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:48:36.112617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.112596 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-5b7d44546c-s2b66"] Nov 03 16:48:36.206696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.206641 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" podStartSLOduration=2.206626591 podStartE2EDuration="2.206626591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:36.169844285 +0000 UTC m=+2011.503057868" watchObservedRunningTime="2026-11-03 16:48:36.206626591 +0000 UTC m=+2011.539840163" Nov 03 16:48:36.208833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.208799 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:48:36.212704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.212676 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-85d4d75bbc-zphwl"] Nov 03 16:48:36.809829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.809788 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df33f83-0636-43c8-b06e-576c8b2b93ae" path="/var/lib/kubelet/pods/1df33f83-0636-43c8-b06e-576c8b2b93ae/volumes" Nov 03 16:48:36.810652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.810625 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2c7db72-924b-4316-ac23-1caf7da1661c" path="/var/lib/kubelet/pods/a2c7db72-924b-4316-ac23-1caf7da1661c/volumes" Nov 03 16:48:36.811676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:36.811503 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e037b66a-5d22-4550-a907-c9d7ad817ec8" path="/var/lib/kubelet/pods/e037b66a-5d22-4550-a907-c9d7ad817ec8/volumes" Nov 03 16:48:41.315518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:41.309015 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:48:41.339478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:41.333077 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:48:43.075869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.075828 6001 generic.go:356] "Generic (PLEG): container finished" podID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerID="c805ad85b7ec3902069a367edb74eb93b9a7e4dab069f2b5e45e76293576d2ef" exitCode=0 Nov 03 16:48:43.076394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.075903 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerDied","Data":"c805ad85b7ec3902069a367edb74eb93b9a7e4dab069f2b5e45e76293576d2ef"} Nov 03 16:48:43.076394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.076372 6001 scope.go:122] "RemoveContainer" containerID="c805ad85b7ec3902069a367edb74eb93b9a7e4dab069f2b5e45e76293576d2ef" Nov 03 16:48:43.077987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.077957 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:43.078074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.078022 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:43.078074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:43.078057 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:48:44.085855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:44.085817 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerStarted","Data":"fb19f5a3d8209299b6872dcea59922f8447690cf5c3fd71cd13a09974fab5a5b"} Nov 03 16:48:44.763470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:44.752824 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:48:44.763470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:44.759587 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 16:48:44.763470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:44.759669 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 16:48:45.763482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:45.762986 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 16:48:45.763482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:45.763060 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 16:48:46.310250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.310204 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" podUID="b4ea57ba-1706-493f-bd7b-f90a5117b655" containerName="payload-processing" containerID="cri-o://02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031" gracePeriod=25 Nov 03 16:48:46.340215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.340163 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" podUID="cab8bcb5-bd34-4797-a266-32cda101e682" containerName="payload-pre-processing" containerID="cri-o://9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742" gracePeriod=25 Nov 03 16:48:46.700341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.700124 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:48:46.784612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.773310 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 16:48:46.784612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.773371 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 16:48:46.791562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.791525 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:48:46.802216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.801667 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\" (UniqueName: \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\") pod \"b4ea57ba-1706-493f-bd7b-f90a5117b655\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " Nov 03 16:48:46.802216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.801820 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\") pod \"b4ea57ba-1706-493f-bd7b-f90a5117b655\" (UID: \"b4ea57ba-1706-493f-bd7b-f90a5117b655\") " Nov 03 16:48:46.802562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.802523 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume" pod "b4ea57ba-1706-493f-bd7b-f90a5117b655" (UID: "b4ea57ba-1706-493f-bd7b-f90a5117b655"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:46.802816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.802781 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4ea57ba-1706-493f-bd7b-f90a5117b655-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.806608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.806531 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj" pod "b4ea57ba-1706-493f-bd7b-f90a5117b655" (UID: "b4ea57ba-1706-493f-bd7b-f90a5117b655"). InnerVolumeSpecName "kube-api-access-hgfwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:46.903891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.903804 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\" (UniqueName: \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\") pod \"cab8bcb5-bd34-4797-a266-32cda101e682\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " Nov 03 16:48:46.904077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.903895 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\") pod \"cab8bcb5-bd34-4797-a266-32cda101e682\" (UID: \"cab8bcb5-bd34-4797-a266-32cda101e682\") " Nov 03 16:48:46.904403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.904376 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume" pod "cab8bcb5-bd34-4797-a266-32cda101e682" (UID: "cab8bcb5-bd34-4797-a266-32cda101e682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:46.904593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.904577 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab8bcb5-bd34-4797-a266-32cda101e682-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.904659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.904595 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfwj\" (UniqueName: \"kubernetes.io/projected/b4ea57ba-1706-493f-bd7b-f90a5117b655-kube-api-access-hgfwj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:46.906759 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:46.906735 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf" pod "cab8bcb5-bd34-4797-a266-32cda101e682" (UID: "cab8bcb5-bd34-4797-a266-32cda101e682"). InnerVolumeSpecName "kube-api-access-t74zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:47.006326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.006292 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74zf\" (UniqueName: \"kubernetes.io/projected/cab8bcb5-bd34-4797-a266-32cda101e682-kube-api-access-t74zf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:48:47.028726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.028697 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:48:47.028899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.028766 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:48:47.108514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.108428 6001 generic.go:356] "Generic (PLEG): container finished" podID="cab8bcb5-bd34-4797-a266-32cda101e682" containerID="9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742" exitCode=0 Nov 03 16:48:47.108688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.108520 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" Nov 03 16:48:47.108688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.108552 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" event={"ID":"cab8bcb5-bd34-4797-a266-32cda101e682","Type":"ContainerDied","Data":"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742"} Nov 03 16:48:47.108688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.108611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc" event={"ID":"cab8bcb5-bd34-4797-a266-32cda101e682","Type":"ContainerDied","Data":"985abd572007726adca967578787f1a20e2f03df53014452fe116d42f179084d"} Nov 03 16:48:47.108688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.108643 6001 scope.go:122] "RemoveContainer" containerID="9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742" Nov 03 16:48:47.110869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.110821 6001 generic.go:356] "Generic (PLEG): container finished" podID="b4ea57ba-1706-493f-bd7b-f90a5117b655" containerID="02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031" exitCode=0 Nov 03 16:48:47.111024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.111005 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" Nov 03 16:48:47.120343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.120308 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" event={"ID":"b4ea57ba-1706-493f-bd7b-f90a5117b655","Type":"ContainerDied","Data":"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031"} Nov 03 16:48:47.120558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.120523 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n" event={"ID":"b4ea57ba-1706-493f-bd7b-f90a5117b655","Type":"ContainerDied","Data":"ce07b238552522aa227305ceb4f07f031811284f2567e646d41fea556bf381d7"} Nov 03 16:48:47.139065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.134766 6001 scope.go:122] "RemoveContainer" containerID="9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742" Nov 03 16:48:47.139065 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:47.135275 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742\": container with ID starting with 9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742 not found: ID does not exist" containerID="9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742" Nov 03 16:48:47.139065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.135307 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742"} err="failed to get container status \"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742\": rpc error: code = NotFound desc = could not find container \"9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742\": container with ID starting with 9ca0676596042a78215d0d7fa730db98002e664bc8b2e10faf2f7dde6d347742 not found: ID does not exist" Nov 03 16:48:47.139065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.135326 6001 scope.go:122] "RemoveContainer" containerID="02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031" Nov 03 16:48:47.189478 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.189414 6001 scope.go:122] "RemoveContainer" containerID="02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031" Nov 03 16:48:47.189914 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:47.189887 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031\": container with ID starting with 02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031 not found: ID does not exist" containerID="02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031" Nov 03 16:48:47.189966 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.189928 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031"} err="failed to get container status \"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031\": rpc error: code = NotFound desc = could not find container \"02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031\": container with ID starting with 02d94d5b1f0cd1dec35a8fe5ad0f40831d1393b0efcfbf8327f8a2929f71e031 not found: ID does not exist" Nov 03 16:48:47.199171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.199135 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:48:47.279242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.279193 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:48:47.282405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.282375 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-64958c9679-8qbwc"] Nov 03 16:48:47.322920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.322868 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:48:47.335717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.335686 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-679cb4bc8c-vk56n"] Nov 03 16:48:47.750915 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.750879 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 16:48:47.751240 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:47.751204 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 16:48:48.575930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.575892 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:48:48.577176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577149 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1df33f83-0636-43c8-b06e-576c8b2b93ae" containerName="payload-processing" Nov 03 16:48:48.577176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577175 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df33f83-0636-43c8-b06e-576c8b2b93ae" containerName="payload-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577200 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b4ea57ba-1706-493f-bd7b-f90a5117b655" containerName="payload-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577208 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ea57ba-1706-493f-bd7b-f90a5117b655" containerName="payload-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577220 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a2c7db72-924b-4316-ac23-1caf7da1661c" containerName="payload-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577227 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2c7db72-924b-4316-ac23-1caf7da1661c" containerName="payload-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577243 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cab8bcb5-bd34-4797-a266-32cda101e682" containerName="payload-pre-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577251 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab8bcb5-bd34-4797-a266-32cda101e682" containerName="payload-pre-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577275 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e037b66a-5d22-4550-a907-c9d7ad817ec8" containerName="payload-pre-processing" Nov 03 16:48:48.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577282 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e037b66a-5d22-4550-a907-c9d7ad817ec8" containerName="payload-pre-processing" Nov 03 16:48:48.577801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577471 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df33f83-0636-43c8-b06e-576c8b2b93ae" containerName="payload-processing" Nov 03 16:48:48.577801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577492 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ea57ba-1706-493f-bd7b-f90a5117b655" containerName="payload-processing" Nov 03 16:48:48.577801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577501 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2c7db72-924b-4316-ac23-1caf7da1661c" containerName="payload-processing" Nov 03 16:48:48.577801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577512 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab8bcb5-bd34-4797-a266-32cda101e682" containerName="payload-pre-processing" Nov 03 16:48:48.577801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.577524 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e037b66a-5d22-4550-a907-c9d7ad817ec8" containerName="payload-pre-processing" Nov 03 16:48:48.582617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.582593 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.597771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.597748 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:48:48.609398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.609368 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d71e12\"" Nov 03 16:48:48.610080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.610062 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d71e12-dockercfg-s2z42\"" Nov 03 16:48:48.684297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.684262 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdj6p\" (UniqueName: \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.684631 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.684608 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.768471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.767829 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 16:48:48.768471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.767893 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 16:48:48.785827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.785789 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.786101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.786076 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdj6p\" (UniqueName: \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.786717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.786691 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.790606 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.790582 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ea57ba-1706-493f-bd7b-f90a5117b655" path="/var/lib/kubelet/pods/b4ea57ba-1706-493f-bd7b-f90a5117b655/volumes" Nov 03 16:48:48.791102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.791081 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab8bcb5-bd34-4797-a266-32cda101e682" path="/var/lib/kubelet/pods/cab8bcb5-bd34-4797-a266-32cda101e682/volumes" Nov 03 16:48:48.810170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.810148 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdj6p\" (UniqueName: \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\") pod \"payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:48.899688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:48.899605 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:49.067008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.066739 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:48:49.079000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.078267 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:48:49.079000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.078420 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.092763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.091436 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkxp\" (UniqueName: \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.092763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.091617 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.193392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.193305 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.193693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.193669 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkxp\" (UniqueName: \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.194049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.194028 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.220660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.220625 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkxp\" (UniqueName: \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\") pod \"payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.242348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.242305 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:48:49.243643 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:48:49.243610 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc62eeb_46dc_4f7d_9058_8c2f4e37d1dc.slice/crio-420b0aa10c5a84b105a764eb73ce5f662ae34b0b38f0d1d0e10b8b59276be8be WatchSource:0}: Error finding container 420b0aa10c5a84b105a764eb73ce5f662ae34b0b38f0d1d0e10b8b59276be8be: Status 404 returned error can't find the container with id 420b0aa10c5a84b105a764eb73ce5f662ae34b0b38f0d1d0e10b8b59276be8be Nov 03 16:48:49.406196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.406161 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:49.742483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.735520 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:48:49.742483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.738897 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:48:49.800594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.747625 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:49.819664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.818258 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d6bb42-dockercfg-vrfkg\"" Nov 03 16:48:49.819664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.818350 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d6bb42\"" Nov 03 16:48:49.828972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.821050 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:48:49.844034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.843945 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:48:49.893464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.888925 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:48:49.915925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.915151 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:49.915925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:49.915339 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpws\" (UniqueName: \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.016384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.016360 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.016588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.016567 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpws\" (UniqueName: \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.017118 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.017100 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.040825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.040799 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpws\" (UniqueName: \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\") pod \"payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.160244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.160201 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" event={"ID":"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc","Type":"ContainerStarted","Data":"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146"} Nov 03 16:48:50.160496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.160300 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" event={"ID":"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc","Type":"ContainerStarted","Data":"420b0aa10c5a84b105a764eb73ce5f662ae34b0b38f0d1d0e10b8b59276be8be"} Nov 03 16:48:50.160496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.160333 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:48:50.162203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.162173 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" event={"ID":"d439fafb-cdae-44f1-a4fd-05937f47f4e3","Type":"ContainerStarted","Data":"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2"} Nov 03 16:48:50.162330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.162223 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" event={"ID":"d439fafb-cdae-44f1-a4fd-05937f47f4e3","Type":"ContainerStarted","Data":"1aec45bbde4e8a7c1045b786c47d51abb8b3cd90f24cdbd02ae077f31451cff3"} Nov 03 16:48:50.162330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.162319 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:48:50.208628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.208574 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" podStartSLOduration=2.208560254 podStartE2EDuration="2.208560254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:50.204118849 +0000 UTC m=+2025.537332418" watchObservedRunningTime="2026-11-03 16:48:50.208560254 +0000 UTC m=+2025.541773824" Nov 03 16:48:50.232228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.232203 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:50.252936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.252881 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" podStartSLOduration=1.2528629150000001 podStartE2EDuration="1.252862915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:50.247608443 +0000 UTC m=+2025.580822013" watchObservedRunningTime="2026-11-03 16:48:50.252862915 +0000 UTC m=+2025.586076485" Nov 03 16:48:50.557139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:50.557108 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:48:50.557749 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:48:50.557723 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec73f99c_5f3d_4b6b_8ac6_f694c29fe1b1.slice/crio-9e91af2180ccd9628b692227c7a44e7fd89f6867511cd10b97872a2b2fe010a5 WatchSource:0}: Error finding container 9e91af2180ccd9628b692227c7a44e7fd89f6867511cd10b97872a2b2fe010a5: Status 404 returned error can't find the container with id 9e91af2180ccd9628b692227c7a44e7fd89f6867511cd10b97872a2b2fe010a5 Nov 03 16:48:51.172298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:51.172260 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" event={"ID":"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1","Type":"ContainerStarted","Data":"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869"} Nov 03 16:48:51.172785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:51.172322 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" event={"ID":"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1","Type":"ContainerStarted","Data":"9e91af2180ccd9628b692227c7a44e7fd89f6867511cd10b97872a2b2fe010a5"} Nov 03 16:48:51.173176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:51.173152 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:48:51.216590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:51.216518 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" podStartSLOduration=2.216499411 podStartE2EDuration="2.216499411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:51.215974266 +0000 UTC m=+2026.549187922" watchObservedRunningTime="2026-11-03 16:48:51.216499411 +0000 UTC m=+2026.549712982" Nov 03 16:48:52.256514 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:48:52.256479 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:48:56.645582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:56.645546 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:48:56.646083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:48:56.645793 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" containerID="cri-o://ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705" gracePeriod=30 Nov 03 16:49:00.947248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:00.947210 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-d66mm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:00.947723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:00.947273 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:01.174795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:01.174764 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:49:01.174980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:01.174847 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:49:01.933380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:01.933297 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:49:02.003341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003305 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.003939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003375 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.003939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003437 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.003939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003620 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.003939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003678 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:02.003939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003864 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.004157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.003979 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.004157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.004013 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.004157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.004062 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.004157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.004072 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:02.004157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.004104 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\") pod \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\" (UID: \"35c1e119-1ffa-47fc-8023-b1d33122dbd5\") " Nov 03 16:49:02.004420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.004394 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:02.006021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.005391 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.006021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.005422 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.006021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.005438 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.006021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.005474 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:02.006021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.005433 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:02.006585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.006545 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:02.006754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.006730 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:02.007531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.007506 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "kube-api-access-9pvws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:02.007979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.007951 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy" pod "35c1e119-1ffa-47fc-8023-b1d33122dbd5" (UID: "35c1e119-1ffa-47fc-8023-b1d33122dbd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:02.107435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107401 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pvws\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-kube-api-access-9pvws\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.107435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107435 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.107435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107462 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.107435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107475 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.107435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107488 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.107861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.107499 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c1e119-1ffa-47fc-8023-b1d33122dbd5-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:02.264861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.264825 6001 generic.go:356] "Generic (PLEG): container finished" podID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerID="ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705" exitCode=0 Nov 03 16:49:02.265080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.264880 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" Nov 03 16:49:02.265080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.264920 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" event={"ID":"35c1e119-1ffa-47fc-8023-b1d33122dbd5","Type":"ContainerDied","Data":"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705"} Nov 03 16:49:02.265080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.264979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm" event={"ID":"35c1e119-1ffa-47fc-8023-b1d33122dbd5","Type":"ContainerDied","Data":"c8baa4e5a7798bcc2d53444aa1ed76600dd616d253c5e2bf1caf89833012a266"} Nov 03 16:49:02.265080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.265004 6001 scope.go:122] "RemoveContainer" containerID="ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705" Nov 03 16:49:02.278021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.277997 6001 scope.go:122] "RemoveContainer" containerID="ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705" Nov 03 16:49:02.278361 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:02.278341 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705\": container with ID starting with ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705 not found: ID does not exist" containerID="ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705" Nov 03 16:49:02.278424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.278371 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705"} err="failed to get container status \"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705\": rpc error: code = NotFound desc = could not find container \"ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705\": container with ID starting with ac4674c0f039a064d10460297ebba1a39bf64f7c98dea18e373a3a0f2c1a3705 not found: ID does not exist" Nov 03 16:49:02.322366 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.322319 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:49:02.333198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.333161 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-d66mm"] Nov 03 16:49:02.791172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:02.791135 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" path="/var/lib/kubelet/pods/35c1e119-1ffa-47fc-8023-b1d33122dbd5/volumes" Nov 03 16:49:03.190172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:03.190093 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:49:04.997277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:04.997248 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jkphl_e1e3af6e-bac6-4c92-b299-f8c669b9a8d9/payload-processing/0.log" Nov 03 16:49:05.231364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.231330 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jkphl_e1e3af6e-bac6-4c92-b299-f8c669b9a8d9/payload-processing/0.log" Nov 03 16:49:05.253976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.253882 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:49:05.254919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.254891 6001 cpu_manager.go:414] "containerMap: removing container" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" Nov 03 16:49:05.254919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.254915 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" Nov 03 16:49:05.255139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.255078 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c1e119-1ffa-47fc-8023-b1d33122dbd5" containerName="istio-proxy" Nov 03 16:49:05.260312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.260268 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.282520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.281581 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:49:05.287638 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.287604 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ad7bf6\"" Nov 03 16:49:05.287841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.287643 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ad7bf6-dockercfg-wv2rc\"" Nov 03 16:49:05.356933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.356896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.357126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.357048 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hctkw\" (UniqueName: \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.434430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.434398 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq_d439fafb-cdae-44f1-a4fd-05937f47f4e3/payload-processing/0.log" Nov 03 16:49:05.435897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.435849 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:49:05.442982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.442281 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.458538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.458506 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.458720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.458631 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hctkw\" (UniqueName: \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.459267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.459241 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.480766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.480733 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:49:05.486152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.486084 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hctkw\" (UniqueName: \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\") pod \"payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.560459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.560353 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.560652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.560627 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49zm\" (UniqueName: \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.584677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.584645 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:05.662156 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.662111 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49zm\" (UniqueName: \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.662411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.662389 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.663491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.663457 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.688748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.688656 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49zm\" (UniqueName: \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\") pod \"payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.759185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.759138 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:05.901672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:05.901639 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:49:06.073834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.073801 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:49:06.074267 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:06.074091 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1125ef8c_f6ce_48fc_9888_b04f7231f30e.slice/crio-c63c9dae7114445ba593179db6755adb9e573bfbe141920a2a78e3ed4a776eb7 WatchSource:0}: Error finding container c63c9dae7114445ba593179db6755adb9e573bfbe141920a2a78e3ed4a776eb7: Status 404 returned error can't find the container with id c63c9dae7114445ba593179db6755adb9e573bfbe141920a2a78e3ed4a776eb7 Nov 03 16:49:06.333103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.333003 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" event={"ID":"204c4df8-bc7d-4237-8643-ad529e1f82a5","Type":"ContainerStarted","Data":"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf"} Nov 03 16:49:06.333103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.333070 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" event={"ID":"204c4df8-bc7d-4237-8643-ad529e1f82a5","Type":"ContainerStarted","Data":"dafbce9a7107f89750879d40d5cfe5917799ca7ce625852175de02f680691c39"} Nov 03 16:49:06.333103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.333103 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:06.334710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.334684 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" event={"ID":"1125ef8c-f6ce-48fc-9888-b04f7231f30e","Type":"ContainerStarted","Data":"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45"} Nov 03 16:49:06.334832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.334730 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" event={"ID":"1125ef8c-f6ce-48fc-9888-b04f7231f30e","Type":"ContainerStarted","Data":"c63c9dae7114445ba593179db6755adb9e573bfbe141920a2a78e3ed4a776eb7"} Nov 03 16:49:06.334832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.334810 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:06.385335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.385268 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" podStartSLOduration=1.385248724 podStartE2EDuration="1.385248724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:06.381924329 +0000 UTC m=+2041.715137898" watchObservedRunningTime="2026-11-03 16:49:06.385248724 +0000 UTC m=+2041.718462298" Nov 03 16:49:06.429104 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:06.429048 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" podStartSLOduration=1.429030982 podStartE2EDuration="1.429030982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:06.428524667 +0000 UTC m=+2041.761738235" watchObservedRunningTime="2026-11-03 16:49:06.429030982 +0000 UTC m=+2041.762244553" Nov 03 16:49:08.887264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:08.887230 6001 scope.go:122] "RemoveContainer" containerID="a125a8ddcc9d0aad627d3973f2ca435802ae7879945878cc25c8819f97a74ce2" Nov 03 16:49:08.898081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:08.898062 6001 scope.go:122] "RemoveContainer" containerID="8808403a19a5970faa22b64254316ffdc12c29642365c3ca853d94997655e0e7" Nov 03 16:49:08.907282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:08.907265 6001 scope.go:122] "RemoveContainer" containerID="6d0d3e91783e1daf955da41eda07f88a987a1bc4171e35e246c76cfab3510ec0" Nov 03 16:49:16.612163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:16.612130 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:49:16.612637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:16.612355 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" podUID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" containerName="payload-processing" containerID="cri-o://0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498" gracePeriod=30 Nov 03 16:49:16.649693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:16.649657 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:49:16.650879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:16.650853 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" podUID="5f755daf-7dc0-4009-b803-8813f631cad0" containerName="payload-pre-processing" containerID="cri-o://e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533" gracePeriod=30 Nov 03 16:49:16.992530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:16.992500 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:49:17.069456 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.069424 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:49:17.163726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.163696 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\" (UniqueName: \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\") pod \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " Nov 03 16:49:17.163899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.163810 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\") pod \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\" (UID: \"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94\") " Nov 03 16:49:17.163946 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.163933 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\") pod \"5f755daf-7dc0-4009-b803-8813f631cad0\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " Nov 03 16:49:17.164026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.164014 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\" (UniqueName: \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\") pod \"5f755daf-7dc0-4009-b803-8813f631cad0\" (UID: \"5f755daf-7dc0-4009-b803-8813f631cad0\") " Nov 03 16:49:17.164332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.164297 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume" pod "a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" (UID: "a8d91eb8-f9c5-4106-8b8a-11023a9a7a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:17.164332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.164319 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume" pod "5f755daf-7dc0-4009-b803-8813f631cad0" (UID: "5f755daf-7dc0-4009-b803-8813f631cad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:17.164738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.164714 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f755daf-7dc0-4009-b803-8813f631cad0-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.164815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.164746 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.166416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.166390 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh" pod "a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" (UID: "a8d91eb8-f9c5-4106-8b8a-11023a9a7a94"). InnerVolumeSpecName "kube-api-access-wf4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:17.166632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.166611 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk" pod "5f755daf-7dc0-4009-b803-8813f631cad0" (UID: "5f755daf-7dc0-4009-b803-8813f631cad0"). InnerVolumeSpecName "kube-api-access-qfbsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:17.270666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.270623 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4bh\" (UniqueName: \"kubernetes.io/projected/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94-kube-api-access-wf4bh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.270666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.270666 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbsk\" (UniqueName: \"kubernetes.io/projected/5f755daf-7dc0-4009-b803-8813f631cad0-kube-api-access-qfbsk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:17.344188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.344159 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:49:17.344378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.344240 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:49:17.422870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.422837 6001 generic.go:356] "Generic (PLEG): container finished" podID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" containerID="0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498" exitCode=0 Nov 03 16:49:17.423066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.422890 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" Nov 03 16:49:17.423066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.423013 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" event={"ID":"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94","Type":"ContainerDied","Data":"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498"} Nov 03 16:49:17.423066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.423061 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj" event={"ID":"a8d91eb8-f9c5-4106-8b8a-11023a9a7a94","Type":"ContainerDied","Data":"4b560ffe5ab8c3eaba15c3349e8e1928a2132d5e890e575a1c27c1b6540a050f"} Nov 03 16:49:17.423251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.423094 6001 scope.go:122] "RemoveContainer" containerID="0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498" Nov 03 16:49:17.425214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.425192 6001 generic.go:356] "Generic (PLEG): container finished" podID="5f755daf-7dc0-4009-b803-8813f631cad0" containerID="e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533" exitCode=0 Nov 03 16:49:17.425335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.425294 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" event={"ID":"5f755daf-7dc0-4009-b803-8813f631cad0","Type":"ContainerDied","Data":"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533"} Nov 03 16:49:17.425397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.425335 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" event={"ID":"5f755daf-7dc0-4009-b803-8813f631cad0","Type":"ContainerDied","Data":"2e8eeec555d55aae6f2a46369b39a33f8234b305d3efc0884130995d1b8aabc0"} Nov 03 16:49:17.425476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.425399 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6" Nov 03 16:49:17.447020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.446976 6001 scope.go:122] "RemoveContainer" containerID="0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498" Nov 03 16:49:17.447451 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:17.447408 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498\": container with ID starting with 0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498 not found: ID does not exist" containerID="0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498" Nov 03 16:49:17.447565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.447465 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498"} err="failed to get container status \"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498\": rpc error: code = NotFound desc = could not find container \"0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498\": container with ID starting with 0d3f704cf19d4c07c36707c9ac3361f528fdd9e945a759d4c426515b5c450498 not found: ID does not exist" Nov 03 16:49:17.447565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.447492 6001 scope.go:122] "RemoveContainer" containerID="e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533" Nov 03 16:49:17.460791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.460773 6001 scope.go:122] "RemoveContainer" containerID="e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533" Nov 03 16:49:17.461112 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:17.461094 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533\": container with ID starting with e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533 not found: ID does not exist" containerID="e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533" Nov 03 16:49:17.461184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.461123 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533"} err="failed to get container status \"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533\": rpc error: code = NotFound desc = could not find container \"e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533\": container with ID starting with e3a5b99fbdf9aae3ad429087c386de1ed9b4f3f6d5557815871805323f18b533 not found: ID does not exist" Nov 03 16:49:17.495919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.495890 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:49:17.500042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.500017 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-624649-5d4cf59f94-85zt6"] Nov 03 16:49:17.535633 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.535520 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:49:17.539358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:17.539332 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-624649-6cbcb9bc44-fznrj"] Nov 03 16:49:18.796822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:18.796786 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f755daf-7dc0-4009-b803-8813f631cad0" path="/var/lib/kubelet/pods/5f755daf-7dc0-4009-b803-8813f631cad0/volumes" Nov 03 16:49:18.797386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:18.797365 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" path="/var/lib/kubelet/pods/a8d91eb8-f9c5-4106-8b8a-11023a9a7a94/volumes" Nov 03 16:49:22.252930 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:22.252890 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:49:27.546390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:27.546347 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:49:27.597120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:27.597086 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:49:32.367175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.367131 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:49:32.367695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.367367 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" containerID="cri-o://5e68d43c9361dcbccb212ef99334933ead2ba1e5beb276a3698acdcd1b5b7cc5" gracePeriod=30 Nov 03 16:49:32.547094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.547049 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" podUID="22bd8306-8599-45f6-994a-9d420b207179" containerName="payload-processing" containerID="cri-o://01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044" gracePeriod=25 Nov 03 16:49:32.598067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.598013 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" podUID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" containerName="payload-pre-processing" containerID="cri-o://e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99" gracePeriod=25 Nov 03 16:49:32.932013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.931980 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:49:32.935687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:32.935663 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:49:33.013018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.012983 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\") pod \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " Nov 03 16:49:33.013209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.013147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\" (UniqueName: \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\") pod \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\" (UID: \"d98ef26e-4b33-45e5-af89-36e2f29af9d2\") " Nov 03 16:49:33.013320 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.013284 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\" (UniqueName: \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\") pod \"22bd8306-8599-45f6-994a-9d420b207179\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " Nov 03 16:49:33.013465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.013419 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume" pod "d98ef26e-4b33-45e5-af89-36e2f29af9d2" (UID: "d98ef26e-4b33-45e5-af89-36e2f29af9d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:33.013601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.013476 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\") pod \"22bd8306-8599-45f6-994a-9d420b207179\" (UID: \"22bd8306-8599-45f6-994a-9d420b207179\") " Nov 03 16:49:33.013919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.013893 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume" pod "22bd8306-8599-45f6-994a-9d420b207179" (UID: "22bd8306-8599-45f6-994a-9d420b207179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:33.014091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.014065 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22bd8306-8599-45f6-994a-9d420b207179-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:33.014215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.014095 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98ef26e-4b33-45e5-af89-36e2f29af9d2-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:33.015714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.015691 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4" pod "22bd8306-8599-45f6-994a-9d420b207179" (UID: "22bd8306-8599-45f6-994a-9d420b207179"). InnerVolumeSpecName "kube-api-access-9t4j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:33.015922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.015899 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2" pod "d98ef26e-4b33-45e5-af89-36e2f29af9d2" (UID: "d98ef26e-4b33-45e5-af89-36e2f29af9d2"). InnerVolumeSpecName "kube-api-access-mskg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:33.116189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.116147 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mskg2\" (UniqueName: \"kubernetes.io/projected/d98ef26e-4b33-45e5-af89-36e2f29af9d2-kube-api-access-mskg2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:33.116189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.116183 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4j4\" (UniqueName: \"kubernetes.io/projected/22bd8306-8599-45f6-994a-9d420b207179-kube-api-access-9t4j4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:33.551383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.551347 6001 generic.go:356] "Generic (PLEG): container finished" podID="22bd8306-8599-45f6-994a-9d420b207179" containerID="01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044" exitCode=0 Nov 03 16:49:33.551914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.551398 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" Nov 03 16:49:33.551914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.551426 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" event={"ID":"22bd8306-8599-45f6-994a-9d420b207179","Type":"ContainerDied","Data":"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044"} Nov 03 16:49:33.551914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.551508 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g" event={"ID":"22bd8306-8599-45f6-994a-9d420b207179","Type":"ContainerDied","Data":"ff730a805421f509b89c070112ceba0602532ef154e6aeda279cfa7a4a8b1eff"} Nov 03 16:49:33.551914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.551533 6001 scope.go:122] "RemoveContainer" containerID="01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044" Nov 03 16:49:33.553232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.553211 6001 generic.go:356] "Generic (PLEG): container finished" podID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" containerID="e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99" exitCode=0 Nov 03 16:49:33.553363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.553260 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" Nov 03 16:49:33.553363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.553273 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" event={"ID":"d98ef26e-4b33-45e5-af89-36e2f29af9d2","Type":"ContainerDied","Data":"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99"} Nov 03 16:49:33.553363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.553320 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x" event={"ID":"d98ef26e-4b33-45e5-af89-36e2f29af9d2","Type":"ContainerDied","Data":"f20d4fdae754fdc06e8bdf2147854db918e743a6a7e7e855052695e94aa00778"} Nov 03 16:49:33.587967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.583118 6001 scope.go:122] "RemoveContainer" containerID="01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044" Nov 03 16:49:33.587967 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:33.584173 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044\": container with ID starting with 01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044 not found: ID does not exist" containerID="01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044" Nov 03 16:49:33.587967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.584208 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044"} err="failed to get container status \"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044\": rpc error: code = NotFound desc = could not find container \"01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044\": container with ID starting with 01ebb62da03fa8a1b8accc0ca56197ef89319a6c8543575b926e0bdf34a3d044 not found: ID does not exist" Nov 03 16:49:33.587967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.584226 6001 scope.go:122] "RemoveContainer" containerID="e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99" Nov 03 16:49:33.598863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.598831 6001 scope.go:122] "RemoveContainer" containerID="e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99" Nov 03 16:49:33.599279 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:33.599249 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99\": container with ID starting with e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99 not found: ID does not exist" containerID="e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99" Nov 03 16:49:33.599372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.599293 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99"} err="failed to get container status \"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99\": rpc error: code = NotFound desc = could not find container \"e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99\": container with ID starting with e120b63fac1ee437644f9f57cd2846ac69251812334c534f4e8442f31b09ed99 not found: ID does not exist" Nov 03 16:49:33.623222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.623180 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:49:33.626802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.626773 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-7ff7c9b58c-9df8g"] Nov 03 16:49:33.663083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.663043 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:49:33.666419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:33.666385 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-54d4b646d7-w9p8x"] Nov 03 16:49:34.795153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:34.795116 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22bd8306-8599-45f6-994a-9d420b207179" path="/var/lib/kubelet/pods/22bd8306-8599-45f6-994a-9d420b207179/volumes" Nov 03 16:49:34.796203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:34.795681 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" path="/var/lib/kubelet/pods/d98ef26e-4b33-45e5-af89-36e2f29af9d2/volumes" Nov 03 16:49:37.035077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.035033 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:49:37.035645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.035319 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" podUID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" containerName="payload-processing" containerID="cri-o://378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2" gracePeriod=30 Nov 03 16:49:37.054243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.054210 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:49:37.054508 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.054468 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" podUID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" containerName="payload-pre-processing" containerID="cri-o://5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146" gracePeriod=30 Nov 03 16:49:37.361374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.361338 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:49:37.526467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.526330 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\" (UniqueName: \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\") pod \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " Nov 03 16:49:37.526467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.526389 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\") pod \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\" (UID: \"d439fafb-cdae-44f1-a4fd-05937f47f4e3\") " Nov 03 16:49:37.530933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.528686 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume" pod "d439fafb-cdae-44f1-a4fd-05937f47f4e3" (UID: "d439fafb-cdae-44f1-a4fd-05937f47f4e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:37.530933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.530865 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp" pod "d439fafb-cdae-44f1-a4fd-05937f47f4e3" (UID: "d439fafb-cdae-44f1-a4fd-05937f47f4e3"). InnerVolumeSpecName "kube-api-access-wgkxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.535277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.535179 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:49:37.592809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.592646 6001 generic.go:356] "Generic (PLEG): container finished" podID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" containerID="378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2" exitCode=0 Nov 03 16:49:37.593171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.593136 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" Nov 03 16:49:37.593285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.593156 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" event={"ID":"d439fafb-cdae-44f1-a4fd-05937f47f4e3","Type":"ContainerDied","Data":"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2"} Nov 03 16:49:37.593285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.593236 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq" event={"ID":"d439fafb-cdae-44f1-a4fd-05937f47f4e3","Type":"ContainerDied","Data":"1aec45bbde4e8a7c1045b786c47d51abb8b3cd90f24cdbd02ae077f31451cff3"} Nov 03 16:49:37.593285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.593275 6001 scope.go:122] "RemoveContainer" containerID="378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2" Nov 03 16:49:37.595976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.595237 6001 generic.go:356] "Generic (PLEG): container finished" podID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" containerID="5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146" exitCode=0 Nov 03 16:49:37.595976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.595371 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" event={"ID":"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc","Type":"ContainerDied","Data":"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146"} Nov 03 16:49:37.595976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.595367 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" Nov 03 16:49:37.595976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.595416 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8" event={"ID":"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc","Type":"ContainerDied","Data":"420b0aa10c5a84b105a764eb73ce5f662ae34b0b38f0d1d0e10b8b59276be8be"} Nov 03 16:49:37.602112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.598820 6001 generic.go:356] "Generic (PLEG): container finished" podID="1795765b-ab43-4308-9277-636688cd2380" containerID="5e68d43c9361dcbccb212ef99334933ead2ba1e5beb276a3698acdcd1b5b7cc5" exitCode=0 Nov 03 16:49:37.602112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.598998 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" event={"ID":"1795765b-ab43-4308-9277-636688cd2380","Type":"ContainerDied","Data":"5e68d43c9361dcbccb212ef99334933ead2ba1e5beb276a3698acdcd1b5b7cc5"} Nov 03 16:49:37.617058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.617032 6001 scope.go:122] "RemoveContainer" containerID="378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2" Nov 03 16:49:37.617641 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:37.617541 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2\": container with ID starting with 378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2 not found: ID does not exist" containerID="378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2" Nov 03 16:49:37.617641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.617582 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2"} err="failed to get container status \"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2\": rpc error: code = NotFound desc = could not find container \"378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2\": container with ID starting with 378023d2c779e595bba52ee0ad138d4023841a0a0807bc8a742b6b2b965170c2 not found: ID does not exist" Nov 03 16:49:37.617641 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.617608 6001 scope.go:122] "RemoveContainer" containerID="5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146" Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.630697 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\" (UniqueName: \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\") pod \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.630844 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\") pod \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\" (UID: \"0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc\") " Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.631514 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkxp\" (UniqueName: \"kubernetes.io/projected/d439fafb-cdae-44f1-a4fd-05937f47f4e3-kube-api-access-wgkxp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.631535 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d439fafb-cdae-44f1-a4fd-05937f47f4e3-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.633563 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume" pod "0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" (UID: "0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:37.635584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.634953 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p" pod "0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" (UID: "0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc"). InnerVolumeSpecName "kube-api-access-zdj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.638403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.638334 6001 scope.go:122] "RemoveContainer" containerID="5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146" Nov 03 16:49:37.649194 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:37.649142 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146\": container with ID starting with 5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146 not found: ID does not exist" containerID="5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146" Nov 03 16:49:37.649329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.649203 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146"} err="failed to get container status \"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146\": rpc error: code = NotFound desc = could not find container \"5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146\": container with ID starting with 5f41c4a56729a68454291ff01e7b56915d74802623c63869617b629ea8a08146 not found: ID does not exist" Nov 03 16:49:37.684035 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.683983 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:49:37.692304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.692256 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d71e12-844446956-tr7wq"] Nov 03 16:49:37.732820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.732780 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.732820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.732816 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdj6p\" (UniqueName: \"kubernetes.io/projected/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc-kube-api-access-zdj6p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.750747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.750720 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:49:37.834024 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.833940 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.834262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.834068 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.834262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.834123 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.834262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.834251 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.834622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.834288 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.834687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.834666 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:37.835105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835070 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:37.835511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835482 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.835621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835526 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.835621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835564 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.835755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835624 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\") pod \"1795765b-ab43-4308-9277-636688cd2380\" (UID: \"1795765b-ab43-4308-9277-636688cd2380\") " Nov 03 16:49:37.835957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835847 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:37.835957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.835863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:37.836105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836082 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:37.836887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836856 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1795765b-ab43-4308-9277-636688cd2380-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.836996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836893 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.836996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836910 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.836996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836926 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.836996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.836943 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.837589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.837563 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:37.837666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.837592 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:37.837822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.837796 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.838808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.838787 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb" pod "1795765b-ab43-4308-9277-636688cd2380" (UID: "1795765b-ab43-4308-9277-636688cd2380"). InnerVolumeSpecName "kube-api-access-sf4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.938566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.938512 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1795765b-ab43-4308-9277-636688cd2380-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.938566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.938546 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf4fb\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-kube-api-access-sf4fb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.938566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.938566 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1795765b-ab43-4308-9277-636688cd2380-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.938566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.938575 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1795765b-ab43-4308-9277-636688cd2380-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.944849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.944813 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:49:37.951195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:37.951164 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d71e12-67488f8774-xjfm8"] Nov 03 16:49:38.608467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.608424 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" Nov 03 16:49:38.608968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.608473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb" event={"ID":"1795765b-ab43-4308-9277-636688cd2380","Type":"ContainerDied","Data":"ca2cd34775a6e7f4ca99d867d52df625c46f87a537f7eaa92d80cba02140baf8"} Nov 03 16:49:38.608968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.608539 6001 scope.go:122] "RemoveContainer" containerID="5e68d43c9361dcbccb212ef99334933ead2ba1e5beb276a3698acdcd1b5b7cc5" Nov 03 16:49:38.677246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.677205 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:49:38.689782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.689749 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-4bljb"] Nov 03 16:49:38.792025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.791991 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" path="/var/lib/kubelet/pods/0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc/volumes" Nov 03 16:49:38.792603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.792580 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1795765b-ab43-4308-9277-636688cd2380" path="/var/lib/kubelet/pods/1795765b-ab43-4308-9277-636688cd2380/volumes" Nov 03 16:49:38.793028 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:38.793008 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" path="/var/lib/kubelet/pods/d439fafb-cdae-44f1-a4fd-05937f47f4e3/volumes" Nov 03 16:49:42.537050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:42.537009 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:49:42.537808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:42.537359 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" containerID="cri-o://fb19f5a3d8209299b6872dcea59922f8447690cf5c3fd71cd13a09974fab5a5b" gracePeriod=30 Nov 03 16:49:44.616722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.616689 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:49:44.617334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617320 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" containerName="payload-pre-processing" Nov 03 16:49:44.617377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617335 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" containerName="payload-pre-processing" Nov 03 16:49:44.617377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617363 6001 cpu_manager.go:414] "containerMap: removing container" podUID="22bd8306-8599-45f6-994a-9d420b207179" containerName="payload-processing" Nov 03 16:49:44.617377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617368 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22bd8306-8599-45f6-994a-9d420b207179" containerName="payload-processing" Nov 03 16:49:44.617377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617376 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" containerName="payload-pre-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617381 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" containerName="payload-pre-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617406 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" containerName="payload-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617411 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" containerName="payload-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617421 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617427 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617436 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5f755daf-7dc0-4009-b803-8813f631cad0" containerName="payload-pre-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617458 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f755daf-7dc0-4009-b803-8813f631cad0" containerName="payload-pre-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617470 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" containerName="payload-processing" Nov 03 16:49:44.617526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617477 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" containerName="payload-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617658 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc62eeb-46dc-4f7d-9058-8c2f4e37d1dc" containerName="payload-pre-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617676 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22bd8306-8599-45f6-994a-9d420b207179" containerName="payload-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617681 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98ef26e-4b33-45e5-af89-36e2f29af9d2" containerName="payload-pre-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617694 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d439fafb-cdae-44f1-a4fd-05937f47f4e3" containerName="payload-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617700 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1795765b-ab43-4308-9277-636688cd2380" containerName="istio-proxy" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617706 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f755daf-7dc0-4009-b803-8813f631cad0" containerName="payload-pre-processing" Nov 03 16:49:44.617803 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.617713 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d91eb8-f9c5-4106-8b8a-11023a9a7a94" containerName="payload-processing" Nov 03 16:49:44.623155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.623131 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.640890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.640861 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-223659-openshift-default-dockercfg-stt5q\"" Nov 03 16:49:44.644798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.644767 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.747728 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.747780 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.747831 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.747881 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.747934 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.748008 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2485\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.748109 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.748160 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.752313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.748200 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.849937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.849894 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850032 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850124 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850191 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850592 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850685 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2485\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850724 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850871 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.850957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.850942 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.851044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.851004 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.852933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.852038 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.852933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.852625 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.852933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.852843 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.853593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.853250 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.855430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.855112 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.855938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.855874 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.876720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.876651 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.876994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.876968 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2485\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\") pod \"e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:44.938263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:44.938224 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:45.257251 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:45.257222 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76604601_96c2_4d3a_8929_e6452881165a.slice/crio-d6a19745a7b75662cba118faa0cdfa2ed93e5717b821c4893a55b67f9dc8a514 WatchSource:0}: Error finding container d6a19745a7b75662cba118faa0cdfa2ed93e5717b821c4893a55b67f9dc8a514: Status 404 returned error can't find the container with id d6a19745a7b75662cba118faa0cdfa2ed93e5717b821c4893a55b67f9dc8a514 Nov 03 16:49:45.258429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.258404 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:49:45.259706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.259671 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:45.259784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.259734 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:45.259784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.259771 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:45.439810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.439777 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:49:45.444567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.444542 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.494499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.490250 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-31de0f-openshift-default-dockercfg-vxx6q\"" Nov 03 16:49:45.494499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.491124 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:49:45.561931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.561831 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.561931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.561896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562009 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562052 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562077 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562198 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562324 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562423 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.562512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.562489 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csv4q\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.668411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.668364 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.670676 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.670846 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.670951 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.671035 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csv4q\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.671227 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.671266 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.671478 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.672463 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.671545 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.683932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.683894 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.684145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.670321 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.684564 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.684540 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.684864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.684838 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.690003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.689975 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.693572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.693541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.695973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.695639 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerStarted","Data":"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc"} Nov 03 16:49:45.695973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.695702 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerStarted","Data":"d6a19745a7b75662cba118faa0cdfa2ed93e5717b821c4893a55b67f9dc8a514"} Nov 03 16:49:45.704805 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.704750 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.724493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.723805 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.724838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.724819 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csv4q\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\") pod \"e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.762692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.762661 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:45.776402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.776347 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podStartSLOduration=1.776328922 podStartE2EDuration="1.776328922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:45.767878903 +0000 UTC m=+2081.101092484" watchObservedRunningTime="2026-11-03 16:49:45.776328922 +0000 UTC m=+2081.109542491" Nov 03 16:49:45.939728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:45.939566 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:46.091954 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:46.091919 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefd5fe3_0bd4_431d_9e9d_a4bde12df0a7.slice/crio-fc38e16df6360be8c49afee4ebb1e0119774d46b0e4352ae4be37a816182e170 WatchSource:0}: Error finding container fc38e16df6360be8c49afee4ebb1e0119774d46b0e4352ae4be37a816182e170: Status 404 returned error can't find the container with id fc38e16df6360be8c49afee4ebb1e0119774d46b0e4352ae4be37a816182e170 Nov 03 16:49:46.092381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.092360 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:49:46.094587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.094553 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:46.094686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.094631 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:46.094686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.094676 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:49:46.704692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.704654 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerStarted","Data":"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e"} Nov 03 16:49:46.705179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.704717 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerStarted","Data":"fc38e16df6360be8c49afee4ebb1e0119774d46b0e4352ae4be37a816182e170"} Nov 03 16:49:46.759137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.759067 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podStartSLOduration=1.759050304 podStartE2EDuration="1.759050304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:46.755822483 +0000 UTC m=+2082.089036053" watchObservedRunningTime="2026-11-03 16:49:46.759050304 +0000 UTC m=+2082.092263874" Nov 03 16:49:46.763325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.763295 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:46.944885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:46.944853 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:47.714714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.714678 6001 generic.go:356] "Generic (PLEG): container finished" podID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerID="fb19f5a3d8209299b6872dcea59922f8447690cf5c3fd71cd13a09974fab5a5b" exitCode=0 Nov 03 16:49:47.715175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.714750 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerDied","Data":"fb19f5a3d8209299b6872dcea59922f8447690cf5c3fd71cd13a09974fab5a5b"} Nov 03 16:49:47.715175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.714821 6001 scope.go:122] "RemoveContainer" containerID="c805ad85b7ec3902069a367edb74eb93b9a7e4dab069f2b5e45e76293576d2ef" Nov 03 16:49:47.715327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.715251 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:47.716318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.716294 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:49:47.782174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.780689 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:47.853487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.853457 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:49:47.907869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.907800 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.908342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.908298 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.908595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.908574 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.908843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.908822 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.909065 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.909024 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.909402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.909384 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.909672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.909653 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.909998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.909978 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.910141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.910125 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\") pod \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\" (UID: \"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d\") " Nov 03 16:49:47.918071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.918034 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:47.918852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.918824 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:47.919756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.919727 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:47.924563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.924520 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:47.934996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.934931 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:47.935357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.935326 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:47.936008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.935977 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:47.940174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.940143 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "kube-api-access-zjfhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:47.943961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:47.940414 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token" pod "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" (UID: "70cf8f64-79d8-4ec9-8ce5-b6616f7de71d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:48.013185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013144 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013178 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013187 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013200 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013212 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013220 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013228 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjfhg\" (UniqueName: \"kubernetes.io/projected/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-kube-api-access-zjfhg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013238 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.013432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.013247 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:49:48.723645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.723615 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" Nov 03 16:49:48.724167 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.723611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk" event={"ID":"70cf8f64-79d8-4ec9-8ce5-b6616f7de71d","Type":"ContainerDied","Data":"db8b5fdbb7eef0577101f2f97ae196234e0be1abf0d7349da37c22845e151620"} Nov 03 16:49:48.724167 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.723768 6001 scope.go:122] "RemoveContainer" containerID="fb19f5a3d8209299b6872dcea59922f8447690cf5c3fd71cd13a09974fab5a5b" Nov 03 16:49:48.724288 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.724277 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:48.725480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.725430 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:49:48.797599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.797568 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:49:48.810148 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:48.810111 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d71e12-openshift-default-85c6dfbb9c-l9pkk"] Nov 03 16:49:50.794178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.794142 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" path="/var/lib/kubelet/pods/70cf8f64-79d8-4ec9-8ce5-b6616f7de71d/volumes" Nov 03 16:49:50.864376 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.864337 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:49:50.865069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.865052 6001 cpu_manager.go:414] "containerMap: removing container" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.865122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.865070 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.865335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.865317 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.870893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.870871 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:50.889212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.889172 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:49:50.897171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.897141 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-223659\"" Nov 03 16:49:50.897299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.897183 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-223659-dockercfg-5fp2g\"" Nov 03 16:49:50.904821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.904789 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:49:50.905926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.905900 6001 cpu_manager.go:414] "containerMap: removing container" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.905926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.905925 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.906198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.906092 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="70cf8f64-79d8-4ec9-8ce5-b6616f7de71d" containerName="istio-proxy" Nov 03 16:49:50.910262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.910241 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:50.929295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.929261 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:49:50.966756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.966720 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn5cz\" (UniqueName: \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:50.966970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:50.966950 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.068323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.068225 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.068565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.068326 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.068565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.068498 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn5cz\" (UniqueName: \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.068565 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.068529 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7b2\" (UniqueName: \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.069090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.069064 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.092395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.092366 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn5cz\" (UniqueName: \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\") pod \"payload-processing-e2e-praxis-a-223659-5b495f6945-n754g\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.169963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.169928 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.170135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.170086 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7b2\" (UniqueName: \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.170811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.170787 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.186870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.186835 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.194121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.194097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7b2\" (UniqueName: \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\") pod \"payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.230992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.230955 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.495332 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:51.495300 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99796687_d3af_4e6a_b86e_b957520e8f38.slice/crio-adaaee970005f09c05dd028a1f4bf09dfff9d7325eabbf050831bc84cbf3f8b8 WatchSource:0}: Error finding container adaaee970005f09c05dd028a1f4bf09dfff9d7325eabbf050831bc84cbf3f8b8: Status 404 returned error can't find the container with id adaaee970005f09c05dd028a1f4bf09dfff9d7325eabbf050831bc84cbf3f8b8 Nov 03 16:49:51.495491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.495415 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:49:51.598316 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:51.598239 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56583617_d568_4ced_a838_35fe0af7a2b8.slice/crio-e3357c0a1bd6cf334ece3125057f606300cd84fbdaaacb706dbabd23ac6c7776 WatchSource:0}: Error finding container e3357c0a1bd6cf334ece3125057f606300cd84fbdaaacb706dbabd23ac6c7776: Status 404 returned error can't find the container with id e3357c0a1bd6cf334ece3125057f606300cd84fbdaaacb706dbabd23ac6c7776 Nov 03 16:49:51.598527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.598502 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:49:51.757099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.757051 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" event={"ID":"56583617-d568-4ced-a838-35fe0af7a2b8","Type":"ContainerStarted","Data":"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31"} Nov 03 16:49:51.757295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.757160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" event={"ID":"56583617-d568-4ced-a838-35fe0af7a2b8","Type":"ContainerStarted","Data":"e3357c0a1bd6cf334ece3125057f606300cd84fbdaaacb706dbabd23ac6c7776"} Nov 03 16:49:51.757295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.757196 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:49:51.759205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.759160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" event={"ID":"99796687-d3af-4e6a-b86e-b957520e8f38","Type":"ContainerStarted","Data":"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc"} Nov 03 16:49:51.759339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.759220 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" event={"ID":"99796687-d3af-4e6a-b86e-b957520e8f38","Type":"ContainerStarted","Data":"adaaee970005f09c05dd028a1f4bf09dfff9d7325eabbf050831bc84cbf3f8b8"} Nov 03 16:49:51.759339 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.759245 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:49:51.800227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.800176 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" podStartSLOduration=1.80016329 podStartE2EDuration="1.80016329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:51.79946911 +0000 UTC m=+2087.132682670" watchObservedRunningTime="2026-11-03 16:49:51.80016329 +0000 UTC m=+2087.133376859" Nov 03 16:49:51.845350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:51.845288 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" podStartSLOduration=1.845272138 podStartE2EDuration="1.845272138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:51.842059697 +0000 UTC m=+2087.175273266" watchObservedRunningTime="2026-11-03 16:49:51.845272138 +0000 UTC m=+2087.178485706" Nov 03 16:49:52.256656 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:49:52.256610 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:49:52.419096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.419045 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:49:52.423590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.423558 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.441135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.441107 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-31de0f\"" Nov 03 16:49:52.441798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.441291 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-31de0f-dockercfg-q9d7m\"" Nov 03 16:49:52.444703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.444683 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:49:52.584424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.584341 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:49:52.596749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.595731 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.602644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.602594 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsbr\" (UniqueName: \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.602855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.602832 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.608733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.608702 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:49:52.704059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.704024 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.704217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.704074 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzjf\" (UniqueName: \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.704217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.704171 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsbr\" (UniqueName: \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.704326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.704308 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.704734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.704714 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.728688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.728661 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsbr\" (UniqueName: \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\") pod \"payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.739063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.739031 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:52.806712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.806496 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.806712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.806621 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzjf\" (UniqueName: \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.807811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.807747 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.831869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.831829 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzjf\" (UniqueName: \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\") pod \"payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:52.919005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:52.918918 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:53.053398 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:53.053365 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf705464_963d_40aa_9793_2902d6f3b916.slice/crio-ee02631c9dbfdfbec1997d2ff1b9a2cdaa79a0890d1457adb572708f5feeb98b WatchSource:0}: Error finding container ee02631c9dbfdfbec1997d2ff1b9a2cdaa79a0890d1457adb572708f5feeb98b: Status 404 returned error can't find the container with id ee02631c9dbfdfbec1997d2ff1b9a2cdaa79a0890d1457adb572708f5feeb98b Nov 03 16:49:53.056589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.056559 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:49:53.221504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.221435 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:49:53.221905 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:49:53.221882 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78d3d66_7bdc_44b4_b73b_d5278133c966.slice/crio-51b4efce7ba9a82f53187f0f9e572b181b84a6323f856faab477dbd954abd17b WatchSource:0}: Error finding container 51b4efce7ba9a82f53187f0f9e572b181b84a6323f856faab477dbd954abd17b: Status 404 returned error can't find the container with id 51b4efce7ba9a82f53187f0f9e572b181b84a6323f856faab477dbd954abd17b Nov 03 16:49:53.779355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.779306 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" event={"ID":"df705464-963d-40aa-9793-2902d6f3b916","Type":"ContainerStarted","Data":"8c48c77c034927e0cf769939b67080a8e4dfe3cc5003fe1bc403993a00b2e201"} Nov 03 16:49:53.779778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.779758 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:49:53.779868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.779806 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" event={"ID":"df705464-963d-40aa-9793-2902d6f3b916","Type":"ContainerStarted","Data":"ee02631c9dbfdfbec1997d2ff1b9a2cdaa79a0890d1457adb572708f5feeb98b"} Nov 03 16:49:53.781741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.781710 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" event={"ID":"f78d3d66-7bdc-44b4-b73b-d5278133c966","Type":"ContainerStarted","Data":"9e0efec1572eae62c4e7b65bfbe290407280d21f56f8ba63282b338ae57e3e30"} Nov 03 16:49:53.781901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.781758 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" event={"ID":"f78d3d66-7bdc-44b4-b73b-d5278133c966","Type":"ContainerStarted","Data":"51b4efce7ba9a82f53187f0f9e572b181b84a6323f856faab477dbd954abd17b"} Nov 03 16:49:53.781901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.781786 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:49:53.824757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.824702 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" podStartSLOduration=1.824688168 podStartE2EDuration="1.824688168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:53.822501873 +0000 UTC m=+2089.155715453" watchObservedRunningTime="2026-11-03 16:49:53.824688168 +0000 UTC m=+2089.157901787" Nov 03 16:49:53.867905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:49:53.867840 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" podStartSLOduration=1.8678206670000002 podStartE2EDuration="1.867820667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:53.864704337 +0000 UTC m=+2089.197917905" watchObservedRunningTime="2026-11-03 16:49:53.867820667 +0000 UTC m=+2089.201034236" Nov 03 16:50:02.769475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:02.769409 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:50:02.769982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:02.769531 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:50:04.792936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:04.792911 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:50:04.793395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:04.792967 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:50:05.111372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.111284 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:50:05.123904 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.123878 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:50:05.373075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.372997 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:50:05.378058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.378031 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.400001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.399970 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:50:05.406789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.406766 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-549271-openshift-default-dockercfg-fdcm4\"" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490393 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490572 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490629 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490657 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490703 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjs6\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.490777 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.494040 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.494468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.494169 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.495047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.494543 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.597370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.597329 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.597727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.597698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.597896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.597877 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598001 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598109 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598244 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598230 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjs6\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598396 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598382 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598528 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598672 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.598958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598933 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.599112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.599084 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.599183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.599183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.598419 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.599736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.599539 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.602880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.602854 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.617140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.617105 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.630691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.630533 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjs6\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.634500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.634412 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\") pod \"e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:05.699789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:05.699751 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:06.027501 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:50:06.027439 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04670ce1_1dc0_4373_9db0_707cfa924865.slice/crio-d699fc1257b74cb8d19ef75608d686c3d41900037c557380514fdddc7bd56da6 WatchSource:0}: Error finding container d699fc1257b74cb8d19ef75608d686c3d41900037c557380514fdddc7bd56da6: Status 404 returned error can't find the container with id d699fc1257b74cb8d19ef75608d686c3d41900037c557380514fdddc7bd56da6 Nov 03 16:50:06.028269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.028243 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:50:06.030574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.030537 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:06.030646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.030609 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:06.030646 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.030640 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:06.888432 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.888391 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerStarted","Data":"2ccb91e9c6d42bc7daa809d7f67e86e968e53bde39e790290e1e915f0da19003"} Nov 03 16:50:06.888645 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.888480 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerStarted","Data":"d699fc1257b74cb8d19ef75608d686c3d41900037c557380514fdddc7bd56da6"} Nov 03 16:50:06.951581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:06.951525 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podStartSLOduration=1.9515071929999999 podStartE2EDuration="1.951507193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:06.95035791 +0000 UTC m=+2102.283571479" watchObservedRunningTime="2026-11-03 16:50:06.951507193 +0000 UTC m=+2102.284720762" Nov 03 16:50:07.700743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:07.700709 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:07.706102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:07.706076 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:07.896507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:07.896479 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:07.897402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:07.897383 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:50:08.974226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:08.974196 6001 scope.go:122] "RemoveContainer" containerID="f4c9b6e655c7fce33d2419da42958b8d0fa05d99df119a132142356e00f1d7da" Nov 03 16:50:08.984501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:08.984477 6001 scope.go:122] "RemoveContainer" containerID="cc7e5c5aae2afd6f5e443baa133901fa556049f0eb3dc2e2d8f3184a835ab7a3" Nov 03 16:50:11.689225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.689188 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:50:11.695066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.695045 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.710999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.710962 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-549271\"" Nov 03 16:50:11.723901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.723880 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-549271-dockercfg-d58cp\"" Nov 03 16:50:11.730574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.730540 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:50:11.818477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.791251 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgntq\" (UniqueName: \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.819183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.819149 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.922657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.922609 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgntq\" (UniqueName: \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.923131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.923099 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.926563 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.926498 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:11.950083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:11.950014 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgntq\" (UniqueName: \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\") pod \"payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:12.011169 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.011124 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:12.316503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.316465 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:50:12.317001 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:50:12.316965 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445a0142_54af_4027_97b1_6792794b0f6a.slice/crio-b4f9a35e5c5d8bbe19ba0aa267e2be040a4a8f0c89fbfa677401a677987641eb WatchSource:0}: Error finding container b4f9a35e5c5d8bbe19ba0aa267e2be040a4a8f0c89fbfa677401a677987641eb: Status 404 returned error can't find the container with id b4f9a35e5c5d8bbe19ba0aa267e2be040a4a8f0c89fbfa677401a677987641eb Nov 03 16:50:12.946499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.946427 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" event={"ID":"445a0142-54af-4027-97b1-6792794b0f6a","Type":"ContainerStarted","Data":"445584088c33de926e6e2baa61b1bdc4c5cb5889c87b1ac86251a4f06dd4fb8a"} Nov 03 16:50:12.946924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.946528 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:12.946924 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.946561 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" event={"ID":"445a0142-54af-4027-97b1-6792794b0f6a","Type":"ContainerStarted","Data":"b4f9a35e5c5d8bbe19ba0aa267e2be040a4a8f0c89fbfa677401a677987641eb"} Nov 03 16:50:12.992562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:12.992506 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" podStartSLOduration=1.992491536 podStartE2EDuration="1.992491536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:12.990660745 +0000 UTC m=+2108.323874313" watchObservedRunningTime="2026-11-03 16:50:12.992491536 +0000 UTC m=+2108.325705105" Nov 03 16:50:22.251950 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:50:22.251907 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:50:23.956514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:23.956469 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:50:45.952662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:45.952625 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:50:45.959580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:45.958688 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:45.988890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:45.988859 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:50:45.996143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:45.996095 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f2923b-openshift-default-dockercfg-gz76n\"" Nov 03 16:50:46.051850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.051743 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.051850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.051797 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtb7j\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.051931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.051982 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052227 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.052213 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.052270 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.052310 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.052348 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.052405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.052381 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153439 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153548 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153807 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153852 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153868 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153904 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.153995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.153927 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154289 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154314 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154341 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154414 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154461 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtb7j\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154674 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.154797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.154755 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.156737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.156710 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.157845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.157815 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.178420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.178383 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.178903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.178883 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtb7j\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\") pod \"e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.313392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.313358 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:46.696265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.696233 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:50:46.696978 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:50:46.696948 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d883dbb_b29b_47d0_acb8_c61080ac989b.slice/crio-828fff1d1dc9acc5dd17de81bc81a60e15b3b0811c500afd2dfb15500605f5ed WatchSource:0}: Error finding container 828fff1d1dc9acc5dd17de81bc81a60e15b3b0811c500afd2dfb15500605f5ed: Status 404 returned error can't find the container with id 828fff1d1dc9acc5dd17de81bc81a60e15b3b0811c500afd2dfb15500605f5ed Nov 03 16:50:46.699761 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.699731 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:46.699838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.699794 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:46.699838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:46.699821 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:47.322012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:47.321398 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerStarted","Data":"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69"} Nov 03 16:50:47.322012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:47.321465 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerStarted","Data":"828fff1d1dc9acc5dd17de81bc81a60e15b3b0811c500afd2dfb15500605f5ed"} Nov 03 16:50:47.387861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:47.387797 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podStartSLOduration=2.387779201 podStartE2EDuration="2.387779201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:47.38307222 +0000 UTC m=+2142.716285792" watchObservedRunningTime="2026-11-03 16:50:47.387779201 +0000 UTC m=+2142.720992771" Nov 03 16:50:48.321612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.321571 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:48.329100 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.329071 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:48.330961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.330934 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:48.331903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.331884 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:50:48.756570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.756535 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:50:48.773204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:48.756611 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:50:49.184809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.184512 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": read tcp 10.134.0.2:57908->10.134.0.179:15021: read: connection reset by peer" start-of-body= Nov 03 16:50:49.184809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.184586 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": read tcp 10.134.0.2:57908->10.134.0.179:15021: read: connection reset by peer" Nov 03 16:50:49.339487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.339458 6001 generic.go:356] "Generic (PLEG): container finished" podID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerID="9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e" exitCode=0 Nov 03 16:50:49.339871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.339533 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerDied","Data":"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e"} Nov 03 16:50:49.339943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.339925 6001 scope.go:122] "RemoveContainer" containerID="9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e" Nov 03 16:50:49.341320 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341300 6001 generic.go:356] "Generic (PLEG): container finished" podID="76604601-96c2-4d3a-8929-e6452881165a" containerID="b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc" exitCode=0 Nov 03 16:50:49.341424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341322 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerDied","Data":"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc"} Nov 03 16:50:49.341726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341701 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:49.341780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341768 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:49.341824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341779 6001 scope.go:122] "RemoveContainer" containerID="b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc" Nov 03 16:50:49.341824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.341801 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:49.343153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.343121 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:49.343258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.343184 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:49.343258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:49.343218 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:50:50.391469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.375770 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerStarted","Data":"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931"} Nov 03 16:50:50.391469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.388865 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerStarted","Data":"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f"} Nov 03 16:50:50.770469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.763836 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:50:50.770469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.764321 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:50.770469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.764381 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:50.938644 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.938614 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:50:50.939206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.939132 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:50.939397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:50.939378 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:51.801700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:51.801657 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:51.802367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:51.802312 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:51.956846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:51.956383 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:51.956846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:51.956467 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:52.007493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:51.976060 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:50:52.109343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.109258 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:50:52.109760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.109689 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.127769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.127743 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f2923b\"" Nov 03 16:50:52.127943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.127808 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f2923b-dockercfg-frp8t\"" Nov 03 16:50:52.303292 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:50:52.303248 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:50:52.304571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.304538 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4dx\" (UniqueName: \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.304866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.304848 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.406347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.406255 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.406555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.406532 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4dx\" (UniqueName: \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.407102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.407075 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.431158 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.431132 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4dx\" (UniqueName: \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\") pod \"payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.431534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.431513 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:52.767470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.763189 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:52.767470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.763250 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:52.816500 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.816418 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:50:52.940802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.940757 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:52.941005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:52.940851 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:53.420666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.420620 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" event={"ID":"be85e2c3-de19-4235-90a8-f0673d2cb7cf","Type":"ContainerStarted","Data":"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd"} Nov 03 16:50:53.420880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.420683 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:50:53.420880 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.420701 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" event={"ID":"be85e2c3-de19-4235-90a8-f0673d2cb7cf","Type":"ContainerStarted","Data":"b14440cf2af4b318d964d925f97e51dcc7e6166481b5fb0804b9b9b6bb5cc5d2"} Nov 03 16:50:53.486471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.465516 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" podStartSLOduration=2.465495977 podStartE2EDuration="2.465495977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:53.463913679 +0000 UTC m=+2148.797127251" watchObservedRunningTime="2026-11-03 16:50:53.465495977 +0000 UTC m=+2148.798709549" Nov 03 16:50:53.770467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.765580 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:53.770467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.765647 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:53.813735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.813702 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:50:53.939174 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.939138 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:53.939622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:53.939194 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:54.764723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:54.764682 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:54.765062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:54.765027 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:54.938753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:54.938714 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:54.938919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:54.938772 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:55.765470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:55.763176 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Nov 03 16:50:55.765470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:55.763243 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Nov 03 16:50:55.939719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:55.939683 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Nov 03 16:50:55.939920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:55.939749 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Nov 03 16:50:56.895468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.888407 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:50:56.914062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.914036 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:50:56.920625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.920597 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:50:56.957056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.957019 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:50:56.959046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.958319 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:50:56.970238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:56.970034 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:50:58.814874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:58.814805 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" podUID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" containerName="payload-processing" containerID="cri-o://7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869" gracePeriod=25 Nov 03 16:50:59.181258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.181221 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:50:59.202717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.202686 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\" (UniqueName: \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\") pod \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " Nov 03 16:50:59.202892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.202795 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\") pod \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\" (UID: \"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1\") " Nov 03 16:50:59.203175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.203138 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume" pod "ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" (UID: "ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:59.203385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.203364 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:50:59.204869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.204846 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws" pod "ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" (UID: "ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1"). InnerVolumeSpecName "kube-api-access-kqpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:59.304931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.304890 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpws\" (UniqueName: \"kubernetes.io/projected/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1-kube-api-access-kqpws\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:50:59.559469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.553927 6001 generic.go:356] "Generic (PLEG): container finished" podID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" containerID="7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869" exitCode=0 Nov 03 16:50:59.559469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.554170 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" event={"ID":"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1","Type":"ContainerDied","Data":"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869"} Nov 03 16:50:59.559469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.554225 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" event={"ID":"ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1","Type":"ContainerDied","Data":"9e91af2180ccd9628b692227c7a44e7fd89f6867511cd10b97872a2b2fe010a5"} Nov 03 16:50:59.559469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.554258 6001 scope.go:122] "RemoveContainer" containerID="7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869" Nov 03 16:50:59.559469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.554428 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr" Nov 03 16:50:59.580947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.580923 6001 scope.go:122] "RemoveContainer" containerID="7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869" Nov 03 16:50:59.581314 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:50:59.581291 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869\": container with ID starting with 7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869 not found: ID does not exist" containerID="7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869" Nov 03 16:50:59.581364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.581324 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869"} err="failed to get container status \"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869\": rpc error: code = NotFound desc = could not find container \"7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869\": container with ID starting with 7414c8a52b1bfff400ad6a040c88eb4e75283499aaa1d1088a6c25a727623869 not found: ID does not exist" Nov 03 16:50:59.623745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.623702 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:50:59.636687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:50:59.636656 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d6bb42-6b478cfb95-r52gr"] Nov 03 16:51:00.794083 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:00.794049 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" path="/var/lib/kubelet/pods/ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1/volumes" Nov 03 16:51:02.302939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:02.302884 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5_1321a470-df83-4378-ad62-fbb74b56a207/payload-processing/0.log" Nov 03 16:51:02.521843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:02.521804 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jkphl_e1e3af6e-bac6-4c92-b299-f8c669b9a8d9/payload-processing/0.log" Nov 03 16:51:02.752461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:02.752418 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-jkphl_e1e3af6e-bac6-4c92-b299-f8c669b9a8d9/payload-processing/0.log" Nov 03 16:51:02.943439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:02.943356 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8_1125ef8c-f6ce-48fc-9888-b04f7231f30e/payload-processing/0.log" Nov 03 16:51:03.129578 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:03.129549 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8_1125ef8c-f6ce-48fc-9888-b04f7231f30e/payload-processing/0.log" Nov 03 16:51:04.343746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:04.343696 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:51:04.344269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:04.343969 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" containerID="cri-o://b41f9a3fec6c17fe4a8bddd899fc2d63d630108c3d5322566bd69eda3be581e7" gracePeriod=30 Nov 03 16:51:04.445117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:04.445085 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:51:05.414038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:05.413991 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:51:06.625935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.625828 6001 generic.go:356] "Generic (PLEG): container finished" podID="04670ce1-1dc0-4373-9db0-707cfa924865" containerID="2ccb91e9c6d42bc7daa809d7f67e86e968e53bde39e790290e1e915f0da19003" exitCode=0 Nov 03 16:51:06.625935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.625908 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerDied","Data":"2ccb91e9c6d42bc7daa809d7f67e86e968e53bde39e790290e1e915f0da19003"} Nov 03 16:51:06.626742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.626390 6001 scope.go:122] "RemoveContainer" containerID="2ccb91e9c6d42bc7daa809d7f67e86e968e53bde39e790290e1e915f0da19003" Nov 03 16:51:06.628323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.628268 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:06.628405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.628367 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:06.628484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:06.628404 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:07.636251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:07.636212 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerStarted","Data":"17133c3d6944424f17f345001606cd4f27839b2af6ff90ecfc94a83ec97b9e4e"} Nov 03 16:51:07.700430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:07.700391 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:51:07.702267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:07.702214 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:07.702440 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:07.702307 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:07.897085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:07.896995 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:51:08.700732 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:08.700692 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:08.701195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:08.700768 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:09.682669 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.682635 6001 generic.go:356] "Generic (PLEG): container finished" podID="0642b25b-8bc6-4918-b666-359d42813315" containerID="b41f9a3fec6c17fe4a8bddd899fc2d63d630108c3d5322566bd69eda3be581e7" exitCode=0 Nov 03 16:51:09.682827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.682783 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:51:09.682827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.682817 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" event={"ID":"0642b25b-8bc6-4918-b666-359d42813315","Type":"ContainerDied","Data":"b41f9a3fec6c17fe4a8bddd899fc2d63d630108c3d5322566bd69eda3be581e7"} Nov 03 16:51:09.682903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.682867 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" event={"ID":"0642b25b-8bc6-4918-b666-359d42813315","Type":"ContainerDied","Data":"7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf"} Nov 03 16:51:09.682903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.682891 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ab4d015f3fec38b0c6e4e5aaf0b646709bb0d65e5d75ff4094c61dd4b5987bf" Nov 03 16:51:09.701107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.701053 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:09.701637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.701130 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:09.807595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807558 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.807595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807598 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.807958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807620 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.807958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807862 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.807958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807933 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.807958 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807934 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:09.808181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807963 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.808181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.807998 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:09.808181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808023 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.808181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.808181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808165 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:09.808403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808188 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\") pod \"0642b25b-8bc6-4918-b666-359d42813315\" (UID: \"0642b25b-8bc6-4918-b666-359d42813315\") " Nov 03 16:51:09.808403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808236 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:09.808895 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.808758 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:09.809145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.809128 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.809239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.809147 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.809239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.809156 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.809239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.809165 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0642b25b-8bc6-4918-b666-359d42813315-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.809239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.809174 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.810919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.810888 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:09.811093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.811071 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "kube-api-access-tq9lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:09.811175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.811160 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:09.811651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.811634 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo" pod "0642b25b-8bc6-4918-b666-359d42813315" (UID: "0642b25b-8bc6-4918-b666-359d42813315"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:09.911072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.910975 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0642b25b-8bc6-4918-b666-359d42813315-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.911072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.911012 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0642b25b-8bc6-4918-b666-359d42813315-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.911072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.911026 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq9lf\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-kube-api-access-tq9lf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:09.911072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:09.911040 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0642b25b-8bc6-4918-b666-359d42813315-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:10.415120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.415071 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" podUID="445a0142-54af-4027-97b1-6792794b0f6a" containerName="payload-pre-processing" containerID="cri-o://445584088c33de926e6e2baa61b1bdc4c5cb5889c87b1ac86251a4f06dd4fb8a" gracePeriod=25 Nov 03 16:51:10.701165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.700676 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:10.701165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.700737 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:10.709485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.706761 6001 generic.go:356] "Generic (PLEG): container finished" podID="445a0142-54af-4027-97b1-6792794b0f6a" containerID="445584088c33de926e6e2baa61b1bdc4c5cb5889c87b1ac86251a4f06dd4fb8a" exitCode=0 Nov 03 16:51:10.709485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.706920 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h" Nov 03 16:51:10.709485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.706983 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" event={"ID":"445a0142-54af-4027-97b1-6792794b0f6a","Type":"ContainerDied","Data":"445584088c33de926e6e2baa61b1bdc4c5cb5889c87b1ac86251a4f06dd4fb8a"} Nov 03 16:51:10.798561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.798531 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:51:10.814626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.814587 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:51:10.821467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.821407 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rmj4h"] Nov 03 16:51:10.821887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.821862 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\" (UniqueName: \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\") pod \"445a0142-54af-4027-97b1-6792794b0f6a\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " Nov 03 16:51:10.822142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.822124 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\") pod \"445a0142-54af-4027-97b1-6792794b0f6a\" (UID: \"445a0142-54af-4027-97b1-6792794b0f6a\") " Nov 03 16:51:10.822619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.822595 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume" pod "445a0142-54af-4027-97b1-6792794b0f6a" (UID: "445a0142-54af-4027-97b1-6792794b0f6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:10.824210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.824184 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq" pod "445a0142-54af-4027-97b1-6792794b0f6a" (UID: "445a0142-54af-4027-97b1-6792794b0f6a"). InnerVolumeSpecName "kube-api-access-cgntq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:10.924436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.924402 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgntq\" (UniqueName: \"kubernetes.io/projected/445a0142-54af-4027-97b1-6792794b0f6a-kube-api-access-cgntq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:10.924679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:10.924665 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/445a0142-54af-4027-97b1-6792794b0f6a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:11.700380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.700339 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:11.700620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.700396 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:11.715926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.715893 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" Nov 03 16:51:11.715926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.715908 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg" event={"ID":"445a0142-54af-4027-97b1-6792794b0f6a","Type":"ContainerDied","Data":"b4f9a35e5c5d8bbe19ba0aa267e2be040a4a8f0c89fbfa677401a677987641eb"} Nov 03 16:51:11.716467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.716001 6001 scope.go:122] "RemoveContainer" containerID="445584088c33de926e6e2baa61b1bdc4c5cb5889c87b1ac86251a4f06dd4fb8a" Nov 03 16:51:11.790940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.790903 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:51:11.795435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:11.795401 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-856c6799f5-ctsjg"] Nov 03 16:51:12.700486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:12.700433 6001 patch_prober.go:29] interesting pod/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Nov 03 16:51:12.700675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:12.700508 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Nov 03 16:51:12.792352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:12.792299 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0642b25b-8bc6-4918-b666-359d42813315" path="/var/lib/kubelet/pods/0642b25b-8bc6-4918-b666-359d42813315/volumes" Nov 03 16:51:12.792936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:12.792822 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445a0142-54af-4027-97b1-6792794b0f6a" path="/var/lib/kubelet/pods/445a0142-54af-4027-97b1-6792794b0f6a/volumes" Nov 03 16:51:13.706329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:13.706277 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:51:13.707675 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:13.707645 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:51:22.252671 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:51:22.252578 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:51:33.861928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:33.861885 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:51:33.896430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:33.896389 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:51:38.880592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:38.880514 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" podUID="f78d3d66-7bdc-44b4-b73b-d5278133c966" containerName="payload-processing" containerID="cri-o://9e0efec1572eae62c4e7b65bfbe290407280d21f56f8ba63282b338ae57e3e30" gracePeriod=25 Nov 03 16:51:38.906994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:38.903956 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" podUID="df705464-963d-40aa-9793-2902d6f3b916" containerName="payload-pre-processing" containerID="cri-o://8c48c77c034927e0cf769939b67080a8e4dfe3cc5003fe1bc403993a00b2e201" gracePeriod=25 Nov 03 16:51:39.040205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.040168 6001 generic.go:356] "Generic (PLEG): container finished" podID="f78d3d66-7bdc-44b4-b73b-d5278133c966" containerID="9e0efec1572eae62c4e7b65bfbe290407280d21f56f8ba63282b338ae57e3e30" exitCode=0 Nov 03 16:51:39.040427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.040405 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" event={"ID":"f78d3d66-7bdc-44b4-b73b-d5278133c966","Type":"ContainerDied","Data":"9e0efec1572eae62c4e7b65bfbe290407280d21f56f8ba63282b338ae57e3e30"} Nov 03 16:51:39.042506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.042389 6001 generic.go:356] "Generic (PLEG): container finished" podID="df705464-963d-40aa-9793-2902d6f3b916" containerID="8c48c77c034927e0cf769939b67080a8e4dfe3cc5003fe1bc403993a00b2e201" exitCode=0 Nov 03 16:51:39.042506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.042424 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" event={"ID":"df705464-963d-40aa-9793-2902d6f3b916","Type":"ContainerDied","Data":"8c48c77c034927e0cf769939b67080a8e4dfe3cc5003fe1bc403993a00b2e201"} Nov 03 16:51:39.229279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.227875 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:51:39.247716 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.247690 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:51:39.330368 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.330322 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\" (UniqueName: \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\") pod \"f78d3d66-7bdc-44b4-b73b-d5278133c966\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " Nov 03 16:51:39.330601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.330475 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\") pod \"df705464-963d-40aa-9793-2902d6f3b916\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " Nov 03 16:51:39.330751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.330728 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\" (UniqueName: \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\") pod \"df705464-963d-40aa-9793-2902d6f3b916\" (UID: \"df705464-963d-40aa-9793-2902d6f3b916\") " Nov 03 16:51:39.330834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.330797 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\") pod \"f78d3d66-7bdc-44b4-b73b-d5278133c966\" (UID: \"f78d3d66-7bdc-44b4-b73b-d5278133c966\") " Nov 03 16:51:39.332099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.332066 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume" pod "f78d3d66-7bdc-44b4-b73b-d5278133c966" (UID: "f78d3d66-7bdc-44b4-b73b-d5278133c966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:39.332237 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.332102 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume" pod "df705464-963d-40aa-9793-2902d6f3b916" (UID: "df705464-963d-40aa-9793-2902d6f3b916"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:39.334797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.334769 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr" pod "df705464-963d-40aa-9793-2902d6f3b916" (UID: "df705464-963d-40aa-9793-2902d6f3b916"). InnerVolumeSpecName "kube-api-access-fjsbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:39.335384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.335356 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf" pod "f78d3d66-7bdc-44b4-b73b-d5278133c966" (UID: "f78d3d66-7bdc-44b4-b73b-d5278133c966"). InnerVolumeSpecName "kube-api-access-jnzjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:39.433542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.433486 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df705464-963d-40aa-9793-2902d6f3b916-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:39.433542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.433532 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjsbr\" (UniqueName: \"kubernetes.io/projected/df705464-963d-40aa-9793-2902d6f3b916-kube-api-access-fjsbr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:39.433542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.433543 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f78d3d66-7bdc-44b4-b73b-d5278133c966-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:39.433542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:39.433552 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzjf\" (UniqueName: \"kubernetes.io/projected/f78d3d66-7bdc-44b4-b73b-d5278133c966-kube-api-access-jnzjf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:40.050129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.050097 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" Nov 03 16:51:40.050708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.050137 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq" event={"ID":"df705464-963d-40aa-9793-2902d6f3b916","Type":"ContainerDied","Data":"ee02631c9dbfdfbec1997d2ff1b9a2cdaa79a0890d1457adb572708f5feeb98b"} Nov 03 16:51:40.050708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.050205 6001 scope.go:122] "RemoveContainer" containerID="8c48c77c034927e0cf769939b67080a8e4dfe3cc5003fe1bc403993a00b2e201" Nov 03 16:51:40.052130 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.052110 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" Nov 03 16:51:40.052257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.052147 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd" event={"ID":"f78d3d66-7bdc-44b4-b73b-d5278133c966","Type":"ContainerDied","Data":"51b4efce7ba9a82f53187f0f9e572b181b84a6323f856faab477dbd954abd17b"} Nov 03 16:51:40.066523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.066468 6001 scope.go:122] "RemoveContainer" containerID="9e0efec1572eae62c4e7b65bfbe290407280d21f56f8ba63282b338ae57e3e30" Nov 03 16:51:40.109662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.109626 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:51:40.117103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.117067 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-31de0f-549f6dcf6c-xlzrd"] Nov 03 16:51:40.152228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.152188 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:51:40.156113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.156070 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31de0f-bf4664f5-wxmlq"] Nov 03 16:51:40.792596 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.792560 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df705464-963d-40aa-9793-2902d6f3b916" path="/var/lib/kubelet/pods/df705464-963d-40aa-9793-2902d6f3b916/volumes" Nov 03 16:51:40.793052 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:40.793038 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78d3d66-7bdc-44b4-b73b-d5278133c966" path="/var/lib/kubelet/pods/f78d3d66-7bdc-44b4-b73b-d5278133c966/volumes" Nov 03 16:51:41.513650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.513612 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:51:41.514138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.513907 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" containerID="cri-o://f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f" gracePeriod=30 Nov 03 16:51:41.818325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.818234 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw"] Nov 03 16:51:41.819017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.818998 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" containerName="payload-processing" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819018 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" containerName="payload-processing" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819039 6001 cpu_manager.go:414] "containerMap: removing container" podUID="df705464-963d-40aa-9793-2902d6f3b916" containerName="payload-pre-processing" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819045 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df705464-963d-40aa-9793-2902d6f3b916" containerName="payload-pre-processing" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819072 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819076 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" Nov 03 16:51:41.819099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819097 6001 cpu_manager.go:414] "containerMap: removing container" podUID="445a0142-54af-4027-97b1-6792794b0f6a" containerName="payload-pre-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819105 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445a0142-54af-4027-97b1-6792794b0f6a" containerName="payload-pre-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819126 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f78d3d66-7bdc-44b4-b73b-d5278133c966" containerName="payload-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819135 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78d3d66-7bdc-44b4-b73b-d5278133c966" containerName="payload-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819284 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78d3d66-7bdc-44b4-b73b-d5278133c966" containerName="payload-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819302 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec73f99c-5f3d-4b6b-8ac6-f694c29fe1b1" containerName="payload-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819311 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="df705464-963d-40aa-9793-2902d6f3b916" containerName="payload-pre-processing" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819321 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0642b25b-8bc6-4918-b666-359d42813315" containerName="istio-proxy" Nov 03 16:51:41.819384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.819331 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="445a0142-54af-4027-97b1-6792794b0f6a" containerName="payload-pre-processing" Nov 03 16:51:41.822913 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.822893 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:41.839154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.839100 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:51:41.839386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.839362 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:51:41.839523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.839437 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:51:41.839689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.839666 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:51:41.852483 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.843476 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw"] Nov 03 16:51:41.920594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.919812 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:51:41.920594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.919876 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:51:41.987920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.987870 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qkn\" (UniqueName: \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:41.988266 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.988241 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:41.988364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:41.988293 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.089274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.089172 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.089274 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.089217 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.089612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.089277 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qkn\" (UniqueName: \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.090080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.090056 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.091788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.091761 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.113827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.113792 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qkn\" (UniqueName: \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\") pod \"maas-api-revoke-sub-27a19b17-7npmw\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.137689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.137636 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:42.491765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.490138 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:51:42.496250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.496214 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:51:42.724329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:42.719958 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw"] Nov 03 16:51:43.104712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:43.104666 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" event={"ID":"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60","Type":"ContainerStarted","Data":"d481c6fe8741436912dc25747c117e89eb3f44f3d40f01662c910349421ea5f6"} Nov 03 16:51:44.122899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:44.122866 6001 generic.go:356] "Generic (PLEG): container finished" podID="5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" containerID="a9430da21e5580cbc7bf715aa906b04ca779b768648001e23ab1a16876857056" exitCode=0 Nov 03 16:51:44.123316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:44.122949 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" event={"ID":"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60","Type":"ContainerDied","Data":"a9430da21e5580cbc7bf715aa906b04ca779b768648001e23ab1a16876857056"} Nov 03 16:51:45.340737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.340703 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:45.436538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.436435 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\") pod \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " Nov 03 16:51:45.436714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.436546 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\") pod \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " Nov 03 16:51:45.436714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.436690 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\" (UniqueName: \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\") pod \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\" (UID: \"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60\") " Nov 03 16:51:45.437054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.437015 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca" pod "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" (UID: "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:45.437406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.437383 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:45.439365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.439324 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn" pod "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" (UID: "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60"). InnerVolumeSpecName "kube-api-access-82qkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:45.439503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.439348 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca" pod "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" (UID: "5d83d6a7-cfbe-45a8-bcbb-c09692eafd60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:45.539385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.539339 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:45.539385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:45.539378 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qkn\" (UniqueName: \"kubernetes.io/projected/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60-kube-api-access-82qkn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.184489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.181674 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" event={"ID":"5d83d6a7-cfbe-45a8-bcbb-c09692eafd60","Type":"ContainerDied","Data":"d481c6fe8741436912dc25747c117e89eb3f44f3d40f01662c910349421ea5f6"} Nov 03 16:51:46.184489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.181726 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d481c6fe8741436912dc25747c117e89eb3f44f3d40f01662c910349421ea5f6" Nov 03 16:51:46.184489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.181783 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw" Nov 03 16:51:46.810738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.810712 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:51:46.892951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.892910 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893066 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893103 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893193 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893377 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893406 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893458 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.893643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.893577 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\") pod \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\" (UID: \"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7\") " Nov 03 16:51:46.894925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.894896 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:46.895410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.895387 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:46.896248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.896092 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:46.896663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.896386 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:46.900654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.900614 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:46.913498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.913469 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:46.913610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.913587 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:46.914426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.914384 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:46.915225 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.915198 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q" pod "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" (UID: "aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7"). InnerVolumeSpecName "kube-api-access-csv4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:46.995602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995564 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995597 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995606 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995614 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995622 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995630 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995638 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csv4q\" (UniqueName: \"kubernetes.io/projected/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-kube-api-access-csv4q\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995647 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:46.995830 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:46.995656 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:47.052920 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.052844 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg"] Nov 03 16:51:47.053584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053566 6001 cpu_manager.go:414] "containerMap: removing container" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:47.053584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053585 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:47.053753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053609 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" containerName="revoke-keys" Nov 03 16:51:47.053753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053614 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" containerName="revoke-keys" Nov 03 16:51:47.053753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053741 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" containerName="revoke-keys" Nov 03 16:51:47.053850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.053757 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:47.056406 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.056391 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.074082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.074056 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:51:47.074512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.074492 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:51:47.075527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.074947 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:51:47.075527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.075009 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:51:47.080297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.080265 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg"] Nov 03 16:51:47.152138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.152094 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.152337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.152189 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.157392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.154800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrclw\" (UniqueName: \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.194988 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.194954 6001 generic.go:356] "Generic (PLEG): container finished" podID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerID="f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f" exitCode=0 Nov 03 16:51:47.195147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.195126 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerDied","Data":"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f"} Nov 03 16:51:47.195239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.195178 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" event={"ID":"aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7","Type":"ContainerDied","Data":"fc38e16df6360be8c49afee4ebb1e0119774d46b0e4352ae4be37a816182e170"} Nov 03 16:51:47.195239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.195209 6001 scope.go:122] "RemoveContainer" containerID="f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f" Nov 03 16:51:47.195385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.195369 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54" Nov 03 16:51:47.210753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.210735 6001 scope.go:122] "RemoveContainer" containerID="9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e" Nov 03 16:51:47.220561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.220537 6001 scope.go:122] "RemoveContainer" containerID="f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f" Nov 03 16:51:47.220835 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:51:47.220808 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f\": container with ID starting with f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f not found: ID does not exist" containerID="f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f" Nov 03 16:51:47.220909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.220846 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f"} err="failed to get container status \"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f\": rpc error: code = NotFound desc = could not find container \"f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f\": container with ID starting with f388ab80ebd715cb91b56b2527ac877d457748007606685b2e1bd57834f2c06f not found: ID does not exist" Nov 03 16:51:47.220909 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.220865 6001 scope.go:122] "RemoveContainer" containerID="9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e" Nov 03 16:51:47.221118 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:51:47.221102 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e\": container with ID starting with 9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e not found: ID does not exist" containerID="9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e" Nov 03 16:51:47.221176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.221122 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e"} err="failed to get container status \"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e\": rpc error: code = NotFound desc = could not find container \"9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e\": container with ID starting with 9dbecc1758057e87cbbcb785ff076d016214748a08f03cf6292ed17fdcc1de7e not found: ID does not exist" Nov 03 16:51:47.255946 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.255912 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.256119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.255955 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.256119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.255991 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrclw\" (UniqueName: \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.256740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.256713 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.258674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.258623 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.259648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.259623 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:51:47.263547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.263526 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-31de0f-openshift-default-5cb48db6f7-2rk54"] Nov 03 16:51:47.280866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.280828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrclw\" (UniqueName: \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\") pod \"maas-api-revoke-sub-90456f76-f9qtg\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.369236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.369140 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:47.491430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.491386 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" containerID="cri-o://7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586" gracePeriod=25 Nov 03 16:51:47.500741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.497518 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" containerID="cri-o://002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8" gracePeriod=25 Nov 03 16:51:47.697010 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:51:47.696973 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58e9589_b19b_459c_b432_47ac6b30abd4.slice/crio-bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf WatchSource:0}: Error finding container bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf: Status 404 returned error can't find the container with id bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf Nov 03 16:51:47.699593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:47.699482 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg"] Nov 03 16:51:48.203514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:48.203439 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" event={"ID":"e58e9589-b19b-459c-b432-47ac6b30abd4","Type":"ContainerStarted","Data":"53bc2c93b3aa07c2a8e15ca371cfca4508982a0a1648f0fb4c5818a88cee1afd"} Nov 03 16:51:48.204003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:48.203537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" event={"ID":"e58e9589-b19b-459c-b432-47ac6b30abd4","Type":"ContainerStarted","Data":"bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf"} Nov 03 16:51:48.246697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:48.246626 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" podStartSLOduration=1.246606742 podStartE2EDuration="1.246606742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:48.245999725 +0000 UTC m=+2203.579213296" watchObservedRunningTime="2026-11-03 16:51:48.246606742 +0000 UTC m=+2203.579820311" Nov 03 16:51:48.796626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:48.796581 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" path="/var/lib/kubelet/pods/aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7/volumes" Nov 03 16:51:49.212672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:49.212638 6001 generic.go:356] "Generic (PLEG): container finished" podID="e58e9589-b19b-459c-b432-47ac6b30abd4" containerID="53bc2c93b3aa07c2a8e15ca371cfca4508982a0a1648f0fb4c5818a88cee1afd" exitCode=0 Nov 03 16:51:49.213209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:49.212730 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" event={"ID":"e58e9589-b19b-459c-b432-47ac6b30abd4","Type":"ContainerDied","Data":"53bc2c93b3aa07c2a8e15ca371cfca4508982a0a1648f0fb4c5818a88cee1afd"} Nov 03 16:51:49.371657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:49.371608 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:51:49.371876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:49.371682 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:51:50.395665 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.395637 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:50.507637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.507601 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\" (UniqueName: \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\") pod \"e58e9589-b19b-459c-b432-47ac6b30abd4\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " Nov 03 16:51:50.507848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.507663 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\") pod \"e58e9589-b19b-459c-b432-47ac6b30abd4\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " Nov 03 16:51:50.507995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.507967 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\") pod \"e58e9589-b19b-459c-b432-47ac6b30abd4\" (UID: \"e58e9589-b19b-459c-b432-47ac6b30abd4\") " Nov 03 16:51:50.508179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.508000 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca" pod "e58e9589-b19b-459c-b432-47ac6b30abd4" (UID: "e58e9589-b19b-459c-b432-47ac6b30abd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:50.508600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.508577 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:50.509905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.509877 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw" pod "e58e9589-b19b-459c-b432-47ac6b30abd4" (UID: "e58e9589-b19b-459c-b432-47ac6b30abd4"). InnerVolumeSpecName "kube-api-access-zrclw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:50.510055 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.510033 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca" pod "e58e9589-b19b-459c-b432-47ac6b30abd4" (UID: "e58e9589-b19b-459c-b432-47ac6b30abd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:50.610290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.610249 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e58e9589-b19b-459c-b432-47ac6b30abd4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:50.610290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:50.610292 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrclw\" (UniqueName: \"kubernetes.io/projected/e58e9589-b19b-459c-b432-47ac6b30abd4-kube-api-access-zrclw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:51.228198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.228164 6001 generic.go:356] "Generic (PLEG): container finished" podID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerID="f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69" exitCode=0 Nov 03 16:51:51.228370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.228231 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerDied","Data":"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69"} Nov 03 16:51:51.228743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.228721 6001 scope.go:122] "RemoveContainer" containerID="f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69" Nov 03 16:51:51.229682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.229662 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" event={"ID":"e58e9589-b19b-459c-b432-47ac6b30abd4","Type":"ContainerDied","Data":"bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf"} Nov 03 16:51:51.229757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.229706 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf226de80a2f03785ebbccdd97f5edd50c34987969c7067dfcc1f0fc8150edbf" Nov 03 16:51:51.229757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.229671 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg" Nov 03 16:51:51.230308 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.230281 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:51.230371 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.230348 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:51.230407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.230379 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.789019 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n"] Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.791790 6001 cpu_manager.go:414] "containerMap: removing container" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.791816 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.791870 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e58e9589-b19b-459c-b432-47ac6b30abd4" containerName="revoke-keys" Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.791880 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58e9589-b19b-459c-b432-47ac6b30abd4" containerName="revoke-keys" Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.792028 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aefd5fe3-0bd4-431d-9e9d-a4bde12df0a7" containerName="istio-proxy" Nov 03 16:51:51.792512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.792044 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58e9589-b19b-459c-b432-47ac6b30abd4" containerName="revoke-keys" Nov 03 16:51:51.801807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.801773 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.809616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.809585 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n"] Nov 03 16:51:51.817471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.817403 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:51:51.817630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.817504 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:51:51.817630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.817605 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:51:51.817808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.817653 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:51:51.836344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.836305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.836561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.836470 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cddr8\" (UniqueName: \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.836561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.836555 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.937726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.937687 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.937896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.937769 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cddr8\" (UniqueName: \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.937896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.937821 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.938353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.938324 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.940276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.940254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:51.971194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:51.971153 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cddr8\" (UniqueName: \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\") pod \"maas-api-revoke-sub-cf8b2a94-8km2n\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:52.122975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.122899 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:52.243704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.242807 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerStarted","Data":"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39"} Nov 03 16:51:52.285887 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:51:52.285844 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:51:52.327168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.327132 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:51:52.327434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.327274 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:52.327434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.327315 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:52.473616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:52.473582 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n"] Nov 03 16:51:52.476086 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:51:52.476053 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140cd1c8_3ee4_4920_8806_021f6ef22535.slice/crio-aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d WatchSource:0}: Error finding container aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d: Status 404 returned error can't find the container with id aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d Nov 03 16:51:53.274168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:53.274062 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" event={"ID":"140cd1c8-3ee4-4920-8806-021f6ef22535","Type":"ContainerStarted","Data":"c501a72823eab37a9530cca6539fd4f0a81a2026686530bc3802857529e0d4ce"} Nov 03 16:51:53.274168 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:53.274121 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" event={"ID":"140cd1c8-3ee4-4920-8806-021f6ef22535","Type":"ContainerStarted","Data":"aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d"} Nov 03 16:51:53.317747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:53.317671 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" podStartSLOduration=2.3176558800000002 podStartE2EDuration="2.31765588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:53.316526604 +0000 UTC m=+2208.649740176" watchObservedRunningTime="2026-11-03 16:51:53.31765588 +0000 UTC m=+2208.650869449" Nov 03 16:51:53.321840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:53.321806 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:53.322022 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:53.321864 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:54.283711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:54.283668 6001 generic.go:356] "Generic (PLEG): container finished" podID="140cd1c8-3ee4-4920-8806-021f6ef22535" containerID="c501a72823eab37a9530cca6539fd4f0a81a2026686530bc3802857529e0d4ce" exitCode=0 Nov 03 16:51:54.284128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:54.283751 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" event={"ID":"140cd1c8-3ee4-4920-8806-021f6ef22535","Type":"ContainerDied","Data":"c501a72823eab37a9530cca6539fd4f0a81a2026686530bc3802857529e0d4ce"} Nov 03 16:51:54.321802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:54.321765 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:54.321989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:54.321835 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:55.321585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.321549 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:55.322095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.321613 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:55.494519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.494491 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:55.610936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.610844 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\") pod \"140cd1c8-3ee4-4920-8806-021f6ef22535\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " Nov 03 16:51:55.611124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.610983 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\" (UniqueName: \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\") pod \"140cd1c8-3ee4-4920-8806-021f6ef22535\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " Nov 03 16:51:55.611124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.611047 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\") pod \"140cd1c8-3ee4-4920-8806-021f6ef22535\" (UID: \"140cd1c8-3ee4-4920-8806-021f6ef22535\") " Nov 03 16:51:55.611415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.611389 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca" pod "140cd1c8-3ee4-4920-8806-021f6ef22535" (UID: "140cd1c8-3ee4-4920-8806-021f6ef22535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:55.611748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.611731 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:55.612993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.612968 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8" pod "140cd1c8-3ee4-4920-8806-021f6ef22535" (UID: "140cd1c8-3ee4-4920-8806-021f6ef22535"). InnerVolumeSpecName "kube-api-access-cddr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:55.613103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.612971 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca" pod "140cd1c8-3ee4-4920-8806-021f6ef22535" (UID: "140cd1c8-3ee4-4920-8806-021f6ef22535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:55.713248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.713205 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/140cd1c8-3ee4-4920-8806-021f6ef22535-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:55.713248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:55.713241 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cddr8\" (UniqueName: \"kubernetes.io/projected/140cd1c8-3ee4-4920-8806-021f6ef22535-kube-api-access-cddr8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:51:56.301038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:56.301011 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" Nov 03 16:51:56.301235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:56.301043 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n" event={"ID":"140cd1c8-3ee4-4920-8806-021f6ef22535","Type":"ContainerDied","Data":"aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d"} Nov 03 16:51:56.301235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:56.301095 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aece967f64dc1c7c0d3ad2d94cbbdac60efe12b5b234faf46741b878ca7ebb8d" Nov 03 16:51:56.321276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:56.321249 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:56.321410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:56.321298 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:57.269331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.269294 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:51:57.269847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.269279 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:51:57.321416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.321380 6001 patch_prober.go:29] interesting pod/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Nov 03 16:51:57.321808 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.321759 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Nov 03 16:51:57.748882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.748845 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:51:57.785882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:57.785845 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:51:58.325621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:58.325587 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:51:58.326031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:58.325903 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:51:58.326694 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:51:58.326674 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:52:02.751977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:02.751929 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" containerName="payload-processing" containerID="cri-o://6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc" gracePeriod=25 Nov 03 16:52:02.767847 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:02.767814 6001 logging.go:55] [core] [Channel #4132 SubChannel #4134]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.180:50052: connect: connection refused" Nov 03 16:52:02.787243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:02.787188 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" podUID="56583617-d568-4ced-a838-35fe0af7a2b8" containerName="payload-pre-processing" containerID="cri-o://5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31" gracePeriod=25 Nov 03 16:52:02.918633 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:02.918594 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56583617_d568_4ced_a838_35fe0af7a2b8.slice/crio-conmon-5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:52:03.090048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.090000 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:52:03.119505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.119435 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:52:03.260407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260371 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\" (UniqueName: \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\") pod \"56583617-d568-4ced-a838-35fe0af7a2b8\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " Nov 03 16:52:03.260702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\") pod \"56583617-d568-4ced-a838-35fe0af7a2b8\" (UID: \"56583617-d568-4ced-a838-35fe0af7a2b8\") " Nov 03 16:52:03.260702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260519 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\") pod \"99796687-d3af-4e6a-b86e-b957520e8f38\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " Nov 03 16:52:03.260702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260602 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\" (UniqueName: \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\") pod \"99796687-d3af-4e6a-b86e-b957520e8f38\" (UID: \"99796687-d3af-4e6a-b86e-b957520e8f38\") " Nov 03 16:52:03.260876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260850 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume" pod "56583617-d568-4ced-a838-35fe0af7a2b8" (UID: "56583617-d568-4ced-a838-35fe0af7a2b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:03.260928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.260884 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume" pod "99796687-d3af-4e6a-b86e-b957520e8f38" (UID: "99796687-d3af-4e6a-b86e-b957520e8f38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:03.261301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.261280 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56583617-d568-4ced-a838-35fe0af7a2b8-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:03.261388 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.261306 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99796687-d3af-4e6a-b86e-b957520e8f38-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:03.262753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.262724 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz" pod "99796687-d3af-4e6a-b86e-b957520e8f38" (UID: "99796687-d3af-4e6a-b86e-b957520e8f38"). InnerVolumeSpecName "kube-api-access-fn5cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:03.262855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.262772 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2" pod "56583617-d568-4ced-a838-35fe0af7a2b8" (UID: "56583617-d568-4ced-a838-35fe0af7a2b8"). InnerVolumeSpecName "kube-api-access-zj7b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:03.354063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.353964 6001 generic.go:356] "Generic (PLEG): container finished" podID="56583617-d568-4ced-a838-35fe0af7a2b8" containerID="5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31" exitCode=0 Nov 03 16:52:03.354063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.354012 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" Nov 03 16:52:03.354063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.354054 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" event={"ID":"56583617-d568-4ced-a838-35fe0af7a2b8","Type":"ContainerDied","Data":"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31"} Nov 03 16:52:03.354356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.354117 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk" event={"ID":"56583617-d568-4ced-a838-35fe0af7a2b8","Type":"ContainerDied","Data":"e3357c0a1bd6cf334ece3125057f606300cd84fbdaaacb706dbabd23ac6c7776"} Nov 03 16:52:03.354356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.354141 6001 scope.go:122] "RemoveContainer" containerID="5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31" Nov 03 16:52:03.355881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.355858 6001 generic.go:356] "Generic (PLEG): container finished" podID="99796687-d3af-4e6a-b86e-b957520e8f38" containerID="6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc" exitCode=0 Nov 03 16:52:03.356006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.355909 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" Nov 03 16:52:03.356006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.355945 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" event={"ID":"99796687-d3af-4e6a-b86e-b957520e8f38","Type":"ContainerDied","Data":"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc"} Nov 03 16:52:03.356006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.356004 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" event={"ID":"99796687-d3af-4e6a-b86e-b957520e8f38","Type":"ContainerDied","Data":"adaaee970005f09c05dd028a1f4bf09dfff9d7325eabbf050831bc84cbf3f8b8"} Nov 03 16:52:03.362213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.362190 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn5cz\" (UniqueName: \"kubernetes.io/projected/99796687-d3af-4e6a-b86e-b957520e8f38-kube-api-access-fn5cz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:03.362213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.362212 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7b2\" (UniqueName: \"kubernetes.io/projected/56583617-d568-4ced-a838-35fe0af7a2b8-kube-api-access-zj7b2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:03.368788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.368766 6001 scope.go:122] "RemoveContainer" containerID="5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31" Nov 03 16:52:03.369360 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:03.369210 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31\": container with ID starting with 5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31 not found: ID does not exist" containerID="5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31" Nov 03 16:52:03.369360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.369250 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31"} err="failed to get container status \"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31\": rpc error: code = NotFound desc = could not find container \"5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31\": container with ID starting with 5f591a13837964800899d1cfd8ab4b80bc76260dc4c8526d950502994356af31 not found: ID does not exist" Nov 03 16:52:03.369360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.369274 6001 scope.go:122] "RemoveContainer" containerID="6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc" Nov 03 16:52:03.380111 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.380087 6001 scope.go:122] "RemoveContainer" containerID="6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc" Nov 03 16:52:03.380464 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:03.380425 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc\": container with ID starting with 6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc not found: ID does not exist" containerID="6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc" Nov 03 16:52:03.380537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.380477 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc"} err="failed to get container status \"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc\": rpc error: code = NotFound desc = could not find container \"6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc\": container with ID starting with 6be83b8b8da6cf69630741b34ce7b163cb36c25038b01dcea91b71b8ea27fffc not found: ID does not exist" Nov 03 16:52:03.411146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.411110 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:52:03.415094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.415067 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-223659-7dd8899fd7-cp4hk"] Nov 03 16:52:03.452433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.452396 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:52:03.459131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.459099 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g"] Nov 03 16:52:03.768341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:03.768286 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-223659-5b495f6945-n754g" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded" Nov 03 16:52:04.810841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:04.810808 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56583617-d568-4ced-a838-35fe0af7a2b8" path="/var/lib/kubelet/pods/56583617-d568-4ced-a838-35fe0af7a2b8/volumes" Nov 03 16:52:04.811583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:04.811306 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" path="/var/lib/kubelet/pods/99796687-d3af-4e6a-b86e-b957520e8f38/volumes" Nov 03 16:52:07.247492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.246607 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:52:07.247492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.246722 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:52:07.693793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.693759 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:52:07.721853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.721822 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\") pod \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " Nov 03 16:52:07.722078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.721922 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\" (UniqueName: \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\") pod \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\" (UID: \"4f1c8358-3904-49c1-b3cd-abf9a60b263d\") " Nov 03 16:52:07.722990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.722944 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume" pod "4f1c8358-3904-49c1-b3cd-abf9a60b263d" (UID: "4f1c8358-3904-49c1-b3cd-abf9a60b263d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:07.725045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.725003 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g" pod "4f1c8358-3904-49c1-b3cd-abf9a60b263d" (UID: "4f1c8358-3904-49c1-b3cd-abf9a60b263d"). InnerVolumeSpecName "kube-api-access-p5c9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:07.801081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.801057 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:52:07.824154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.824118 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\") pod \"cc13cb10-885d-4150-90f6-0103e2564505\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " Nov 03 16:52:07.824281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.824235 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\" (UniqueName: \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\") pod \"cc13cb10-885d-4150-90f6-0103e2564505\" (UID: \"cc13cb10-885d-4150-90f6-0103e2564505\") " Nov 03 16:52:07.824666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.824638 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume" pod "cc13cb10-885d-4150-90f6-0103e2564505" (UID: "cc13cb10-885d-4150-90f6-0103e2564505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:07.824748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.824713 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f1c8358-3904-49c1-b3cd-abf9a60b263d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:07.824748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.824731 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5c9g\" (UniqueName: \"kubernetes.io/projected/4f1c8358-3904-49c1-b3cd-abf9a60b263d-kube-api-access-p5c9g\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:07.826407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.826388 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4" pod "cc13cb10-885d-4150-90f6-0103e2564505" (UID: "cc13cb10-885d-4150-90f6-0103e2564505"). InnerVolumeSpecName "kube-api-access-kc8k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:07.926220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.926186 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc13cb10-885d-4150-90f6-0103e2564505-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:07.926220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:07.926214 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8k4\" (UniqueName: \"kubernetes.io/projected/cc13cb10-885d-4150-90f6-0103e2564505-kube-api-access-kc8k4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:08.400389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.400353 6001 generic.go:356] "Generic (PLEG): container finished" podID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerID="7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586" exitCode=0 Nov 03 16:52:08.400943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.400409 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" Nov 03 16:52:08.400943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.400432 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" event={"ID":"4f1c8358-3904-49c1-b3cd-abf9a60b263d","Type":"ContainerDied","Data":"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586"} Nov 03 16:52:08.400943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.400512 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln" event={"ID":"4f1c8358-3904-49c1-b3cd-abf9a60b263d","Type":"ContainerDied","Data":"f36b3615d733fd8518fe1e01f86f4293b6d080140c5a4de59c8c143596f46236"} Nov 03 16:52:08.400943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.400545 6001 scope.go:122] "RemoveContainer" containerID="7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586" Nov 03 16:52:08.401947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.401925 6001 generic.go:356] "Generic (PLEG): container finished" podID="cc13cb10-885d-4150-90f6-0103e2564505" containerID="002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8" exitCode=0 Nov 03 16:52:08.402064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.401968 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" Nov 03 16:52:08.402064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.402009 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" event={"ID":"cc13cb10-885d-4150-90f6-0103e2564505","Type":"ContainerDied","Data":"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8"} Nov 03 16:52:08.402064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.402056 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p" event={"ID":"cc13cb10-885d-4150-90f6-0103e2564505","Type":"ContainerDied","Data":"094e62a23cdc58ed489dd8eb2962c88aa31c236df96ae6e1279fcebcd2d51786"} Nov 03 16:52:08.429392 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.429363 6001 scope.go:122] "RemoveContainer" containerID="7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586" Nov 03 16:52:08.430024 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:08.429989 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586\": container with ID starting with 7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586 not found: ID does not exist" containerID="7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586" Nov 03 16:52:08.430221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.430037 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586"} err="failed to get container status \"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586\": rpc error: code = NotFound desc = could not find container \"7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586\": container with ID starting with 7779406f750a04f2b36c720c0852a581fc05f5c528e714a7e3b1f89576988586 not found: ID does not exist" Nov 03 16:52:08.430221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.430063 6001 scope.go:122] "RemoveContainer" containerID="002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8" Nov 03 16:52:08.442789 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.442763 6001 scope.go:122] "RemoveContainer" containerID="002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8" Nov 03 16:52:08.443148 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:08.443125 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8\": container with ID starting with 002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8 not found: ID does not exist" containerID="002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8" Nov 03 16:52:08.443226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.443160 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8"} err="failed to get container status \"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8\": rpc error: code = NotFound desc = could not find container \"002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8\": container with ID starting with 002ac6eb68d98c3c02ef84ec369e35f0a246c3a79641c192068b547b60a1a1a8 not found: ID does not exist" Nov 03 16:52:08.473576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.473536 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:52:08.478128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.478092 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b9e362-558f89b4f5-js45p"] Nov 03 16:52:08.514642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.514608 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:52:08.517554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.517528 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b9e362-787f56c66f-nmtln"] Nov 03 16:52:08.791505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.791465 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" path="/var/lib/kubelet/pods/4f1c8358-3904-49c1-b3cd-abf9a60b263d/volumes" Nov 03 16:52:08.792050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:08.792027 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc13cb10-885d-4150-90f6-0103e2564505" path="/var/lib/kubelet/pods/cc13cb10-885d-4150-90f6-0103e2564505/volumes" Nov 03 16:52:09.093044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:09.092954 6001 scope.go:122] "RemoveContainer" containerID="b41f9a3fec6c17fe4a8bddd899fc2d63d630108c3d5322566bd69eda3be581e7" Nov 03 16:52:10.331926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.331886 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256"] Nov 03 16:52:10.332658 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332635 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" Nov 03 16:52:10.332752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332659 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" Nov 03 16:52:10.332752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332693 6001 cpu_manager.go:414] "containerMap: removing container" podUID="56583617-d568-4ced-a838-35fe0af7a2b8" containerName="payload-pre-processing" Nov 03 16:52:10.332752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332701 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56583617-d568-4ced-a838-35fe0af7a2b8" containerName="payload-pre-processing" Nov 03 16:52:10.332752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332738 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" Nov 03 16:52:10.332752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332746 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" Nov 03 16:52:10.332917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332762 6001 cpu_manager.go:414] "containerMap: removing container" podUID="140cd1c8-3ee4-4920-8806-021f6ef22535" containerName="revoke-keys" Nov 03 16:52:10.332917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332769 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140cd1c8-3ee4-4920-8806-021f6ef22535" containerName="revoke-keys" Nov 03 16:52:10.332917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332801 6001 cpu_manager.go:414] "containerMap: removing container" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" containerName="payload-processing" Nov 03 16:52:10.332917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332806 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" containerName="payload-processing" Nov 03 16:52:10.333034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332942 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1c8358-3904-49c1-b3cd-abf9a60b263d" containerName="payload-processing" Nov 03 16:52:10.333034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332951 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="56583617-d568-4ced-a838-35fe0af7a2b8" containerName="payload-pre-processing" Nov 03 16:52:10.333034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332963 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc13cb10-885d-4150-90f6-0103e2564505" containerName="payload-pre-processing" Nov 03 16:52:10.333034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332968 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="140cd1c8-3ee4-4920-8806-021f6ef22535" containerName="revoke-keys" Nov 03 16:52:10.333034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.332977 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99796687-d3af-4e6a-b86e-b957520e8f38" containerName="payload-processing" Nov 03 16:52:10.338679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.338659 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.355364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.355335 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256"] Nov 03 16:52:10.365013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.364436 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:52:10.365013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.364479 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:52:10.365013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.364605 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:52:10.365013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.364646 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:52:10.463235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.463196 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.463418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.463249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.463512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.463486 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdq8\" (UniqueName: \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.564609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.564570 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdq8\" (UniqueName: \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.564784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.564719 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.564784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.564758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.565531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.565504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.567214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.567190 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.588984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.588909 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdq8\" (UniqueName: \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\") pod \"maas-api-revoke-sub-32e1621f-nh256\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.654521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.654490 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:10.967232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:10.967203 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256"] Nov 03 16:52:10.967652 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:10.967628 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28e3b40_7747_4e77_957f_c70e80bbe596.slice/crio-0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c WatchSource:0}: Error finding container 0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c: Status 404 returned error can't find the container with id 0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c Nov 03 16:52:11.425144 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:11.425101 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" event={"ID":"c28e3b40-7747-4e77-957f-c70e80bbe596","Type":"ContainerStarted","Data":"dbd85626c3de6da98ce243023e6a753ee7489b34717d2c7f7d90e4ed6a92b666"} Nov 03 16:52:11.425579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:11.425170 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" event={"ID":"c28e3b40-7747-4e77-957f-c70e80bbe596","Type":"ContainerStarted","Data":"0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c"} Nov 03 16:52:11.473720 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:11.473660 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" podStartSLOduration=1.473644369 podStartE2EDuration="1.473644369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:11.472643416 +0000 UTC m=+2226.805856984" watchObservedRunningTime="2026-11-03 16:52:11.473644369 +0000 UTC m=+2226.806857945" Nov 03 16:52:12.434290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:12.434243 6001 generic.go:356] "Generic (PLEG): container finished" podID="c28e3b40-7747-4e77-957f-c70e80bbe596" containerID="dbd85626c3de6da98ce243023e6a753ee7489b34717d2c7f7d90e4ed6a92b666" exitCode=0 Nov 03 16:52:12.434819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:12.434319 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" event={"ID":"c28e3b40-7747-4e77-957f-c70e80bbe596","Type":"ContainerDied","Data":"dbd85626c3de6da98ce243023e6a753ee7489b34717d2c7f7d90e4ed6a92b666"} Nov 03 16:52:13.284403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.284364 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:52:13.284697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.284667 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" podUID="1321a470-df83-4378-ad62-fbb74b56a207" containerName="payload-processing" containerID="cri-o://5fbe8fa871cd668a9bf51eb08b2a6fbd2bc65021a4ab03b57ad36c9ebae82b59" gracePeriod=30 Nov 03 16:52:13.316503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.313101 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:52:13.316503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.313383 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" podUID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" containerName="payload-pre-processing" containerID="cri-o://186064aee239c3bc2a1ca4328bb0b717aa703d21da800c11e4b50c1e67567110" gracePeriod=30 Nov 03 16:52:13.446005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.445963 6001 generic.go:356] "Generic (PLEG): container finished" podID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" containerID="186064aee239c3bc2a1ca4328bb0b717aa703d21da800c11e4b50c1e67567110" exitCode=0 Nov 03 16:52:13.446429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.446035 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" event={"ID":"6e393d06-5b20-44fd-b24e-efa3cc0f0665","Type":"ContainerDied","Data":"186064aee239c3bc2a1ca4328bb0b717aa703d21da800c11e4b50c1e67567110"} Nov 03 16:52:13.448503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.448474 6001 generic.go:356] "Generic (PLEG): container finished" podID="1321a470-df83-4378-ad62-fbb74b56a207" containerID="5fbe8fa871cd668a9bf51eb08b2a6fbd2bc65021a4ab03b57ad36c9ebae82b59" exitCode=0 Nov 03 16:52:13.448664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.448567 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" event={"ID":"1321a470-df83-4378-ad62-fbb74b56a207","Type":"ContainerDied","Data":"5fbe8fa871cd668a9bf51eb08b2a6fbd2bc65021a4ab03b57ad36c9ebae82b59"} Nov 03 16:52:13.812647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.812617 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:52:13.834066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.834037 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:52:13.837498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.837474 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:13.930594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.930545 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\") pod \"c28e3b40-7747-4e77-957f-c70e80bbe596\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " Nov 03 16:52:13.930800 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.930614 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\" (UniqueName: \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\") pod \"1321a470-df83-4378-ad62-fbb74b56a207\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " Nov 03 16:52:13.930875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.930808 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\") pod \"1321a470-df83-4378-ad62-fbb74b56a207\" (UID: \"1321a470-df83-4378-ad62-fbb74b56a207\") " Nov 03 16:52:13.930942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.930913 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\") pod \"c28e3b40-7747-4e77-957f-c70e80bbe596\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " Nov 03 16:52:13.930996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.930955 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\") pod \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " Nov 03 16:52:13.931063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931023 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\" (UniqueName: \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\") pod \"c28e3b40-7747-4e77-957f-c70e80bbe596\" (UID: \"c28e3b40-7747-4e77-957f-c70e80bbe596\") " Nov 03 16:52:13.931115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931091 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\" (UniqueName: \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\") pod \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\" (UID: \"6e393d06-5b20-44fd-b24e-efa3cc0f0665\") " Nov 03 16:52:13.931323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931279 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume" pod "1321a470-df83-4378-ad62-fbb74b56a207" (UID: "1321a470-df83-4378-ad62-fbb74b56a207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:13.931323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931304 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca" pod "c28e3b40-7747-4e77-957f-c70e80bbe596" (UID: "c28e3b40-7747-4e77-957f-c70e80bbe596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:13.931749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931723 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1321a470-df83-4378-ad62-fbb74b56a207-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.931749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931735 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume" pod "6e393d06-5b20-44fd-b24e-efa3cc0f0665" (UID: "6e393d06-5b20-44fd-b24e-efa3cc0f0665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:13.931749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.931751 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:13.933051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.933022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca" pod "c28e3b40-7747-4e77-957f-c70e80bbe596" (UID: "c28e3b40-7747-4e77-957f-c70e80bbe596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:13.933704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.933669 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57" pod "1321a470-df83-4378-ad62-fbb74b56a207" (UID: "1321a470-df83-4378-ad62-fbb74b56a207"). InnerVolumeSpecName "kube-api-access-prm57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:13.933975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.933950 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8" pod "c28e3b40-7747-4e77-957f-c70e80bbe596" (UID: "c28e3b40-7747-4e77-957f-c70e80bbe596"). InnerVolumeSpecName "kube-api-access-zsdq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:13.934069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:13.934016 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p" pod "6e393d06-5b20-44fd-b24e-efa3cc0f0665" (UID: "6e393d06-5b20-44fd-b24e-efa3cc0f0665"). InnerVolumeSpecName "kube-api-access-r946p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:14.033337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.033297 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e393d06-5b20-44fd-b24e-efa3cc0f0665-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:14.033337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.033331 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdq8\" (UniqueName: \"kubernetes.io/projected/c28e3b40-7747-4e77-957f-c70e80bbe596-kube-api-access-zsdq8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:14.033337 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.033342 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r946p\" (UniqueName: \"kubernetes.io/projected/6e393d06-5b20-44fd-b24e-efa3cc0f0665-kube-api-access-r946p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:14.033599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.033353 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c28e3b40-7747-4e77-957f-c70e80bbe596-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:14.033599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.033362 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prm57\" (UniqueName: \"kubernetes.io/projected/1321a470-df83-4378-ad62-fbb74b56a207-kube-api-access-prm57\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:14.456195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.456162 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" Nov 03 16:52:14.456195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.456190 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t" event={"ID":"6e393d06-5b20-44fd-b24e-efa3cc0f0665","Type":"ContainerDied","Data":"447fced1e1e1cfeda65bd516c9d2ad18d7cbb4f056e9390eb16e59525a499a41"} Nov 03 16:52:14.456810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.456249 6001 scope.go:122] "RemoveContainer" containerID="186064aee239c3bc2a1ca4328bb0b717aa703d21da800c11e4b50c1e67567110" Nov 03 16:52:14.457754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.457736 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" Nov 03 16:52:14.457845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.457773 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5" event={"ID":"1321a470-df83-4378-ad62-fbb74b56a207","Type":"ContainerDied","Data":"1c2e1ca9dc80d1a3eeabb9363f46ad284fa70f30017fc2573eaeb106d49cc9c0"} Nov 03 16:52:14.461059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.461039 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" Nov 03 16:52:14.461164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.461082 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256" event={"ID":"c28e3b40-7747-4e77-957f-c70e80bbe596","Type":"ContainerDied","Data":"0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c"} Nov 03 16:52:14.461164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.461135 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0da2a17d9b7afbee5512c9656ee304c8a7ea0e0093d19127ca819344746aa59c" Nov 03 16:52:14.468188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.468166 6001 scope.go:122] "RemoveContainer" containerID="5fbe8fa871cd668a9bf51eb08b2a6fbd2bc65021a4ab03b57ad36c9ebae82b59" Nov 03 16:52:14.517871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.517826 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:52:14.533105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.533073 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-9947ab-6f9c5485c6-pq47t"] Nov 03 16:52:14.560319 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.560232 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:52:14.568975 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.568940 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-9947ab-55bd566786-rfft5"] Nov 03 16:52:14.791376 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.791342 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1321a470-df83-4378-ad62-fbb74b56a207" path="/var/lib/kubelet/pods/1321a470-df83-4378-ad62-fbb74b56a207/volumes" Nov 03 16:52:14.791796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:14.791779 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" path="/var/lib/kubelet/pods/6e393d06-5b20-44fd-b24e-efa3cc0f0665/volumes" Nov 03 16:52:22.251948 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:22.251908 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:52:23.594544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.594502 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57"] Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595324 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1321a470-df83-4378-ad62-fbb74b56a207" containerName="payload-processing" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595344 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1321a470-df83-4378-ad62-fbb74b56a207" containerName="payload-processing" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595359 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" containerName="payload-pre-processing" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595364 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" containerName="payload-pre-processing" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595388 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c28e3b40-7747-4e77-957f-c70e80bbe596" containerName="revoke-keys" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595393 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28e3b40-7747-4e77-957f-c70e80bbe596" containerName="revoke-keys" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595554 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28e3b40-7747-4e77-957f-c70e80bbe596" containerName="revoke-keys" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595589 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1321a470-df83-4378-ad62-fbb74b56a207" containerName="payload-processing" Nov 03 16:52:23.595640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.595601 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e393d06-5b20-44fd-b24e-efa3cc0f0665" containerName="payload-pre-processing" Nov 03 16:52:23.601369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.601344 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.618575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.618535 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:52:23.618575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.618545 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:52:23.620796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.620767 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57"] Nov 03 16:52:23.629755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.629673 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:52:23.631425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.631387 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:52:23.722730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.722693 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtnn7\" (UniqueName: \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.723029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.722833 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.723115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.723061 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.824737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.824691 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.824945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.824860 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.825007 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.824978 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtnn7\" (UniqueName: \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.825627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.825600 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.827204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.827186 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.872056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.863603 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtnn7\" (UniqueName: \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\") pod \"maas-api-revoke-sub-7e9dc695-9xb57\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:23.894222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.894186 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:52:23.894431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.894405 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" containerID="cri-o://6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c" gracePeriod=30 Nov 03 16:52:23.917138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:23.917102 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:24.225696 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:24.225594 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c137a8_f7e8_4326_b3eb_9170945a910f.slice/crio-d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318 WatchSource:0}: Error finding container d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318: Status 404 returned error can't find the container with id d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318 Nov 03 16:52:24.226278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:24.226247 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57"] Nov 03 16:52:24.553847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:24.553815 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" event={"ID":"77c137a8-f7e8-4326-b3eb-9170945a910f","Type":"ContainerStarted","Data":"d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318"} Nov 03 16:52:24.700926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:24.700883 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:52:24.701368 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:24.701095 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" containerID="cri-o://cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931" gracePeriod=30 Nov 03 16:52:25.562677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:25.562633 6001 generic.go:356] "Generic (PLEG): container finished" podID="77c137a8-f7e8-4326-b3eb-9170945a910f" containerID="3e1059ccf8a3822f6486fcfab26a966a04a331f034060d845b195f49c12e7fa4" exitCode=0 Nov 03 16:52:25.562873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:25.562721 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" event={"ID":"77c137a8-f7e8-4326-b3eb-9170945a910f","Type":"ContainerDied","Data":"3e1059ccf8a3822f6486fcfab26a966a04a331f034060d845b195f49c12e7fa4"} Nov 03 16:52:25.844930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:25.844890 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:52:25.845524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:25.845230 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" containerID="cri-o://7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b" gracePeriod=30 Nov 03 16:52:26.742511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.742477 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:26.826696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.826596 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\" (UniqueName: \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\") pod \"77c137a8-f7e8-4326-b3eb-9170945a910f\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " Nov 03 16:52:26.826696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.826640 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\") pod \"77c137a8-f7e8-4326-b3eb-9170945a910f\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " Nov 03 16:52:26.826948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.826905 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\") pod \"77c137a8-f7e8-4326-b3eb-9170945a910f\" (UID: \"77c137a8-f7e8-4326-b3eb-9170945a910f\") " Nov 03 16:52:26.827364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.827331 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca" pod "77c137a8-f7e8-4326-b3eb-9170945a910f" (UID: "77c137a8-f7e8-4326-b3eb-9170945a910f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:26.827523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.827463 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:26.828979 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.828948 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7" pod "77c137a8-f7e8-4326-b3eb-9170945a910f" (UID: "77c137a8-f7e8-4326-b3eb-9170945a910f"). InnerVolumeSpecName "kube-api-access-vtnn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:26.829247 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.829231 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca" pod "77c137a8-f7e8-4326-b3eb-9170945a910f" (UID: "77c137a8-f7e8-4326-b3eb-9170945a910f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:26.929297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.929248 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c137a8-f7e8-4326-b3eb-9170945a910f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:26.929297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.929287 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtnn7\" (UniqueName: \"kubernetes.io/projected/77c137a8-f7e8-4326-b3eb-9170945a910f-kube-api-access-vtnn7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:26.960635 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.960587 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:52:26.960829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:26.960659 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:52:27.581668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:27.581633 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" Nov 03 16:52:27.581873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:27.581633 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57" event={"ID":"77c137a8-f7e8-4326-b3eb-9170945a910f","Type":"ContainerDied","Data":"d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318"} Nov 03 16:52:27.581873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:27.581763 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d202a4a1171dc3365d39d4c7fc1d15ba4f78b3bebe3e8dfe2e998424138fb318" Nov 03 16:52:27.978539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:27.978499 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:52:27.978951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:27.978571 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:52:29.172991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.172915 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:52:29.261228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261195 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261284 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261425 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261378 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261490 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261536 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261609 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261566 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261657 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261758 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.261884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261763 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.261884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261789 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:29.261884 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261830 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\") pod \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\" (UID: \"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7\") " Nov 03 16:52:29.262064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261963 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.262064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.261977 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.262258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262238 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.262572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262543 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.262572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262574 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.262731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262588 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.262731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262599 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.262731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.262611 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.264626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.264595 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:29.264742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.264641 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:29.264742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.264671 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:29.264742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.264721 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk" pod "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" (UID: "10c42d69-d96d-4bf7-87ec-18dc52d1e2b7"). InnerVolumeSpecName "kube-api-access-ml4wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:29.363841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.363804 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.363841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.363836 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.363841 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.363847 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml4wk\" (UniqueName: \"kubernetes.io/projected/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-kube-api-access-ml4wk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.364103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.363856 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:29.598253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.598215 6001 generic.go:356] "Generic (PLEG): container finished" podID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerID="6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c" exitCode=0 Nov 03 16:52:29.598435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.598268 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" Nov 03 16:52:29.598435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.598299 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerDied","Data":"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c"} Nov 03 16:52:29.598435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.598355 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8" event={"ID":"10c42d69-d96d-4bf7-87ec-18dc52d1e2b7","Type":"ContainerDied","Data":"7a26c12de89ea462b976e7a109574e8992f47b8b01d7e35d3f8ee77d6af947f5"} Nov 03 16:52:29.598435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.598380 6001 scope.go:122] "RemoveContainer" containerID="6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c" Nov 03 16:52:29.610746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.610721 6001 scope.go:122] "RemoveContainer" containerID="58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68" Nov 03 16:52:29.620825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.620803 6001 scope.go:122] "RemoveContainer" containerID="6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c" Nov 03 16:52:29.621126 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:29.621107 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c\": container with ID starting with 6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c not found: ID does not exist" containerID="6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c" Nov 03 16:52:29.621166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.621138 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c"} err="failed to get container status \"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c\": rpc error: code = NotFound desc = could not find container \"6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c\": container with ID starting with 6c5083eba267d42014596ce4376f8d25c51dc0edf55cc7ca71805454d0a37c4c not found: ID does not exist" Nov 03 16:52:29.621166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.621160 6001 scope.go:122] "RemoveContainer" containerID="58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68" Nov 03 16:52:29.621418 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:29.621399 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68\": container with ID starting with 58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68 not found: ID does not exist" containerID="58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68" Nov 03 16:52:29.621477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.621425 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68"} err="failed to get container status \"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68\": rpc error: code = NotFound desc = could not find container \"58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68\": container with ID starting with 58445c5eca21e2cece7f59c5d3d5beb26f4bc2cd2f0801a9db545da281365c68 not found: ID does not exist" Nov 03 16:52:29.660208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.660172 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:52:29.672692 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.672641 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b9e362-openshift-default-c78765cf7-6stv8"] Nov 03 16:52:29.874688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.874604 6001 patch_prober.go:29] interesting pod/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:52:29.874688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.874667 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:52:29.998685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:29.998656 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:52:30.072855 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.072814 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.072941 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073134 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:30.073177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073166 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073210 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073245 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073344 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:30.073467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073396 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073437 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073488 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.073651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.073520 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\") pod \"76604601-96c2-4d3a-8929-e6452881165a\" (UID: \"76604601-96c2-4d3a-8929-e6452881165a\") " Nov 03 16:52:30.074383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.074139 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.074383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.074165 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.074604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.074394 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:30.074671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.074621 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:30.074925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.074894 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:30.076748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.076083 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:30.077113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.077085 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:30.077651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.077611 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:30.078614 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.078585 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485" pod "76604601-96c2-4d3a-8929-e6452881165a" (UID: "76604601-96c2-4d3a-8929-e6452881165a"). InnerVolumeSpecName "kube-api-access-l2485". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175183 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175219 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2485\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-kube-api-access-l2485\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175229 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175238 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76604601-96c2-4d3a-8929-e6452881165a-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175247 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76604601-96c2-4d3a-8929-e6452881165a-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175254 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76604601-96c2-4d3a-8929-e6452881165a-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.175271 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.175262 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76604601-96c2-4d3a-8929-e6452881165a-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:30.607479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.607374 6001 generic.go:356] "Generic (PLEG): container finished" podID="76604601-96c2-4d3a-8929-e6452881165a" containerID="cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931" exitCode=0 Nov 03 16:52:30.607479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.607457 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" Nov 03 16:52:30.607726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.607491 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerDied","Data":"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931"} Nov 03 16:52:30.607726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.607581 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4" event={"ID":"76604601-96c2-4d3a-8929-e6452881165a","Type":"ContainerDied","Data":"d6a19745a7b75662cba118faa0cdfa2ed93e5717b821c4893a55b67f9dc8a514"} Nov 03 16:52:30.607726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.607615 6001 scope.go:122] "RemoveContainer" containerID="cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931" Nov 03 16:52:30.619102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.619078 6001 scope.go:122] "RemoveContainer" containerID="b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc" Nov 03 16:52:30.631193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.631169 6001 scope.go:122] "RemoveContainer" containerID="cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931" Nov 03 16:52:30.631575 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:30.631548 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931\": container with ID starting with cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931 not found: ID does not exist" containerID="cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931" Nov 03 16:52:30.631735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.631580 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931"} err="failed to get container status \"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931\": rpc error: code = NotFound desc = could not find container \"cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931\": container with ID starting with cbd33c8d01c2caa2996b14375989d4af884e5d654399cce0bc5ca533331b5931 not found: ID does not exist" Nov 03 16:52:30.631735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.631600 6001 scope.go:122] "RemoveContainer" containerID="b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc" Nov 03 16:52:30.631912 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:30.631891 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc\": container with ID starting with b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc not found: ID does not exist" containerID="b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc" Nov 03 16:52:30.631970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.631920 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc"} err="failed to get container status \"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc\": rpc error: code = NotFound desc = could not find container \"b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc\": container with ID starting with b046496c105a9df46a7ff0df6be34a8bcec069b9f542124e1d0a48a1bc26eefc not found: ID does not exist" Nov 03 16:52:30.666409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.666368 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:52:30.679510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.679432 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-223659-openshift-default-6f96d7d845-pffq4"] Nov 03 16:52:30.791981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.791943 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" path="/var/lib/kubelet/pods/10c42d69-d96d-4bf7-87ec-18dc52d1e2b7/volumes" Nov 03 16:52:30.792545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:30.792528 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76604601-96c2-4d3a-8929-e6452881165a" path="/var/lib/kubelet/pods/76604601-96c2-4d3a-8929-e6452881165a/volumes" Nov 03 16:52:31.143348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.143323 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:52:31.188728 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.188687 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.188818 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.188896 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189068 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189131 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189217 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189258 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189238 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189300 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189309 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.189754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189459 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\") pod \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\" (UID: \"440ab9e3-4d80-4ab2-a596-c6e0e1879499\") " Nov 03 16:52:31.190494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.189766 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:31.190494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.190016 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:31.192216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.192185 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:31.192826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.192775 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "kube-api-access-7sg4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193101 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193277 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193516 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193539 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193554 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sg4s\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-kube-api-access-7sg4s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193569 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193582 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193595 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193608 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.193640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.193620 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.194310 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.194289 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy" pod "440ab9e3-4d80-4ab2-a596-c6e0e1879499" (UID: "440ab9e3-4d80-4ab2-a596-c6e0e1879499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:31.294627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.294579 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/440ab9e3-4d80-4ab2-a596-c6e0e1879499-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:31.618250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.618214 6001 generic.go:356] "Generic (PLEG): container finished" podID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerID="7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b" exitCode=0 Nov 03 16:52:31.618414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.618267 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" Nov 03 16:52:31.618414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.618298 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerDied","Data":"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b"} Nov 03 16:52:31.618414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.618356 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b" event={"ID":"440ab9e3-4d80-4ab2-a596-c6e0e1879499","Type":"ContainerDied","Data":"4c33c71655251d8f614bc5788c000206e345bb82757bdf7b1d6e135a62ff52ad"} Nov 03 16:52:31.618414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.618388 6001 scope.go:122] "RemoveContainer" containerID="7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b" Nov 03 16:52:31.629718 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.629431 6001 scope.go:122] "RemoveContainer" containerID="ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b" Nov 03 16:52:31.634705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.634678 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:52:31.639758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.639739 6001 scope.go:122] "RemoveContainer" containerID="7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b" Nov 03 16:52:31.640040 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:31.640015 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b\": container with ID starting with 7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b not found: ID does not exist" containerID="7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b" Nov 03 16:52:31.640086 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.640052 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b"} err="failed to get container status \"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b\": rpc error: code = NotFound desc = could not find container \"7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b\": container with ID starting with 7ec1e56b9e33f921ed30b0a5123d4ca9541207ca5ac10fe5bb6e60ef5d133a4b not found: ID does not exist" Nov 03 16:52:31.640086 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.640073 6001 scope.go:122] "RemoveContainer" containerID="ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b" Nov 03 16:52:31.640330 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:31.640312 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b\": container with ID starting with ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b not found: ID does not exist" containerID="ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b" Nov 03 16:52:31.640378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.640338 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b"} err="failed to get container status \"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b\": rpc error: code = NotFound desc = could not find container \"ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b\": container with ID starting with ad462a3b359f87a053e5b1a14e6419c1fd616d932a29dea46746990a8c275f1b not found: ID does not exist" Nov 03 16:52:31.676327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.676289 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:52:31.680430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:31.680403 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9947ab-openshift-default-5f584687cf-rpg9b"] Nov 03 16:52:32.791629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:32.791595 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" path="/var/lib/kubelet/pods/440ab9e3-4d80-4ab2-a596-c6e0e1879499/volumes" Nov 03 16:52:36.635082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:36.634991 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" podUID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" containerName="payload-processing" containerID="cri-o://228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd" gracePeriod=25 Nov 03 16:52:36.943963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:36.943931 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:52:37.101950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.101911 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\" (UniqueName: \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\") pod \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " Nov 03 16:52:37.102231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.102035 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\") pod \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\" (UID: \"be85e2c3-de19-4235-90a8-f0673d2cb7cf\") " Nov 03 16:52:37.102726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.102695 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume" pod "be85e2c3-de19-4235-90a8-f0673d2cb7cf" (UID: "be85e2c3-de19-4235-90a8-f0673d2cb7cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:37.104419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.104390 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx" pod "be85e2c3-de19-4235-90a8-f0673d2cb7cf" (UID: "be85e2c3-de19-4235-90a8-f0673d2cb7cf"). InnerVolumeSpecName "kube-api-access-hv4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:37.203896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.203801 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be85e2c3-de19-4235-90a8-f0673d2cb7cf-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:37.203896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.203838 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4dx\" (UniqueName: \"kubernetes.io/projected/be85e2c3-de19-4235-90a8-f0673d2cb7cf-kube-api-access-hv4dx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:37.666106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.666068 6001 generic.go:356] "Generic (PLEG): container finished" podID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" containerID="228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd" exitCode=0 Nov 03 16:52:37.666579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.666118 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" Nov 03 16:52:37.666579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.666174 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" event={"ID":"be85e2c3-de19-4235-90a8-f0673d2cb7cf","Type":"ContainerDied","Data":"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd"} Nov 03 16:52:37.666579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.666228 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m" event={"ID":"be85e2c3-de19-4235-90a8-f0673d2cb7cf","Type":"ContainerDied","Data":"b14440cf2af4b318d964d925f97e51dcc7e6166481b5fb0804b9b9b6bb5cc5d2"} Nov 03 16:52:37.666579 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.666250 6001 scope.go:122] "RemoveContainer" containerID="228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd" Nov 03 16:52:37.676908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.676890 6001 scope.go:122] "RemoveContainer" containerID="228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd" Nov 03 16:52:37.677207 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:37.677186 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd\": container with ID starting with 228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd not found: ID does not exist" containerID="228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd" Nov 03 16:52:37.677254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.677215 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd"} err="failed to get container status \"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd\": rpc error: code = NotFound desc = could not find container \"228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd\": container with ID starting with 228c73a7f61a9c26b743aa1a248906e1c58d05dac047e0bc9ff7895be6dc32cd not found: ID does not exist" Nov 03 16:52:37.722175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.722138 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:52:37.735314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:37.735278 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-f2923b-654dfc9b77-nqd4m"] Nov 03 16:52:38.502163 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502127 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm"] Nov 03 16:52:38.502768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502754 6001 cpu_manager.go:414] "containerMap: removing container" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502769 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502779 6001 cpu_manager.go:414] "containerMap: removing container" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502785 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502796 6001 cpu_manager.go:414] "containerMap: removing container" podUID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" containerName="payload-processing" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502801 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" containerName="payload-processing" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502815 6001 cpu_manager.go:414] "containerMap: removing container" podUID="77c137a8-f7e8-4326-b3eb-9170945a910f" containerName="revoke-keys" Nov 03 16:52:38.502829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502820 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c137a8-f7e8-4326-b3eb-9170945a910f" containerName="revoke-keys" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502852 6001 cpu_manager.go:414] "containerMap: removing container" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502857 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502865 6001 cpu_manager.go:414] "containerMap: removing container" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502871 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502879 6001 cpu_manager.go:414] "containerMap: removing container" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502884 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502896 6001 cpu_manager.go:414] "containerMap: removing container" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502902 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502980 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502987 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.502995 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="440ab9e3-4d80-4ab2-a596-c6e0e1879499" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.503003 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.503013 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="76604601-96c2-4d3a-8929-e6452881165a" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.503020 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c42d69-d96d-4bf7-87ec-18dc52d1e2b7" containerName="istio-proxy" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.503026 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" containerName="payload-processing" Nov 03 16:52:38.503037 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.503033 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c137a8-f7e8-4326-b3eb-9170945a910f" containerName="revoke-keys" Nov 03 16:52:38.507438 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.507416 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.523153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.522868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctvbt\" (UniqueName: \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.523153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.523127 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.523411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.523220 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.526261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.526230 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm"] Nov 03 16:52:38.535673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.535639 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:52:38.535883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.535742 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:52:38.535883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.535816 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:52:38.535883 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.535867 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:52:38.624777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.624734 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.624967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.624823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctvbt\" (UniqueName: \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.624967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.624941 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.625497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.625440 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.627420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.627392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.648910 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.648883 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctvbt\" (UniqueName: \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\") pod \"maas-api-revoke-sub-33243299-h5fhm\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:38.791933 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.791852 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be85e2c3-de19-4235-90a8-f0673d2cb7cf" path="/var/lib/kubelet/pods/be85e2c3-de19-4235-90a8-f0673d2cb7cf/volumes" Nov 03 16:52:38.821642 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:38.821592 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:39.127255 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:39.127218 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc50cff_43a3_4bff_8fb0_b0b648f74427.slice/crio-645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63 WatchSource:0}: Error finding container 645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63: Status 404 returned error can't find the container with id 645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63 Nov 03 16:52:39.128713 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:39.128669 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm"] Nov 03 16:52:39.695567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:39.695516 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" event={"ID":"9bc50cff-43a3-4bff-8fb0-b0b648f74427","Type":"ContainerStarted","Data":"ad6a17fd39bd705811ad05a9a067aea617483c05c644291a011d90d83eea179a"} Nov 03 16:52:39.695917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:39.695599 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" event={"ID":"9bc50cff-43a3-4bff-8fb0-b0b648f74427","Type":"ContainerStarted","Data":"645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63"} Nov 03 16:52:39.743071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:39.743016 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" podStartSLOduration=1.7430021340000001 podStartE2EDuration="1.743002134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:39.738935956 +0000 UTC m=+2255.072149526" watchObservedRunningTime="2026-11-03 16:52:39.743002134 +0000 UTC m=+2255.076215702" Nov 03 16:52:40.704699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:40.704662 6001 generic.go:356] "Generic (PLEG): container finished" podID="9bc50cff-43a3-4bff-8fb0-b0b648f74427" containerID="ad6a17fd39bd705811ad05a9a067aea617483c05c644291a011d90d83eea179a" exitCode=0 Nov 03 16:52:40.705131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:40.704746 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" event={"ID":"9bc50cff-43a3-4bff-8fb0-b0b648f74427","Type":"ContainerDied","Data":"ad6a17fd39bd705811ad05a9a067aea617483c05c644291a011d90d83eea179a"} Nov 03 16:52:41.073263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.073228 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:52:41.073493 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.073434 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" podUID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" containerName="payload-processing" containerID="cri-o://bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45" gracePeriod=30 Nov 03 16:52:41.094393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.094305 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:52:41.094603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.094545 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" podUID="204c4df8-bc7d-4237-8643-ad529e1f82a5" containerName="payload-pre-processing" containerID="cri-o://1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf" gracePeriod=30 Nov 03 16:52:41.475330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.475298 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:52:41.485672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.485641 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:52:41.569421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.569383 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\" (UniqueName: \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\") pod \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " Nov 03 16:52:41.569636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.569502 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\") pod \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\" (UID: \"1125ef8c-f6ce-48fc-9888-b04f7231f30e\") " Nov 03 16:52:41.569636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.569572 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\" (UniqueName: \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\") pod \"204c4df8-bc7d-4237-8643-ad529e1f82a5\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " Nov 03 16:52:41.569636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.569632 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\") pod \"204c4df8-bc7d-4237-8643-ad529e1f82a5\" (UID: \"204c4df8-bc7d-4237-8643-ad529e1f82a5\") " Nov 03 16:52:41.570412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.570149 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume" pod "1125ef8c-f6ce-48fc-9888-b04f7231f30e" (UID: "1125ef8c-f6ce-48fc-9888-b04f7231f30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.570412 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.570363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume" pod "204c4df8-bc7d-4237-8643-ad529e1f82a5" (UID: "204c4df8-bc7d-4237-8643-ad529e1f82a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.572931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.572905 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm" pod "1125ef8c-f6ce-48fc-9888-b04f7231f30e" (UID: "1125ef8c-f6ce-48fc-9888-b04f7231f30e"). InnerVolumeSpecName "kube-api-access-m49zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:41.574743 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.574714 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw" pod "204c4df8-bc7d-4237-8643-ad529e1f82a5" (UID: "204c4df8-bc7d-4237-8643-ad529e1f82a5"). InnerVolumeSpecName "kube-api-access-hctkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:41.671346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.671257 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hctkw\" (UniqueName: \"kubernetes.io/projected/204c4df8-bc7d-4237-8643-ad529e1f82a5-kube-api-access-hctkw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.671346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.671291 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/204c4df8-bc7d-4237-8643-ad529e1f82a5-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.671346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.671301 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m49zm\" (UniqueName: \"kubernetes.io/projected/1125ef8c-f6ce-48fc-9888-b04f7231f30e-kube-api-access-m49zm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.671346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.671310 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1125ef8c-f6ce-48fc-9888-b04f7231f30e-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.713009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.712969 6001 generic.go:356] "Generic (PLEG): container finished" podID="204c4df8-bc7d-4237-8643-ad529e1f82a5" containerID="1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf" exitCode=0 Nov 03 16:52:41.713498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.713021 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" Nov 03 16:52:41.713498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.713052 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" event={"ID":"204c4df8-bc7d-4237-8643-ad529e1f82a5","Type":"ContainerDied","Data":"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf"} Nov 03 16:52:41.713498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.713107 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g" event={"ID":"204c4df8-bc7d-4237-8643-ad529e1f82a5","Type":"ContainerDied","Data":"dafbce9a7107f89750879d40d5cfe5917799ca7ce625852175de02f680691c39"} Nov 03 16:52:41.713498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.713135 6001 scope.go:122] "RemoveContainer" containerID="1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf" Nov 03 16:52:41.714690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.714670 6001 generic.go:356] "Generic (PLEG): container finished" podID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" containerID="bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45" exitCode=0 Nov 03 16:52:41.714799 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.714703 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" event={"ID":"1125ef8c-f6ce-48fc-9888-b04f7231f30e","Type":"ContainerDied","Data":"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45"} Nov 03 16:52:41.714906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.714716 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" Nov 03 16:52:41.715146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.715123 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8" event={"ID":"1125ef8c-f6ce-48fc-9888-b04f7231f30e","Type":"ContainerDied","Data":"c63c9dae7114445ba593179db6755adb9e573bfbe141920a2a78e3ed4a776eb7"} Nov 03 16:52:41.739305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.739174 6001 scope.go:122] "RemoveContainer" containerID="1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf" Nov 03 16:52:41.739770 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:41.739737 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf\": container with ID starting with 1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf not found: ID does not exist" containerID="1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf" Nov 03 16:52:41.739899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.739776 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf"} err="failed to get container status \"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf\": rpc error: code = NotFound desc = could not find container \"1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf\": container with ID starting with 1aa58949c650c3adc4f3da9cdc70ffcae28fee5dc959912641df825e399fa3bf not found: ID does not exist" Nov 03 16:52:41.739899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.739797 6001 scope.go:122] "RemoveContainer" containerID="bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45" Nov 03 16:52:41.752040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.752013 6001 scope.go:122] "RemoveContainer" containerID="bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45" Nov 03 16:52:41.752407 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:41.752385 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45\": container with ID starting with bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45 not found: ID does not exist" containerID="bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45" Nov 03 16:52:41.752535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.752415 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45"} err="failed to get container status \"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45\": rpc error: code = NotFound desc = could not find container \"bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45\": container with ID starting with bbdbc2339b371f83c3a30e78345df36d82e6dbe91a547373033a148867557e45 not found: ID does not exist" Nov 03 16:52:41.789047 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.789016 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:52:41.796350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.796267 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ad7bf6-c9c4fbb9d-xrl6g"] Nov 03 16:52:41.849069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.849035 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:52:41.851607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.851572 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ad7bf6-77f4fc84dd-dmbs8"] Nov 03 16:52:41.904621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.904593 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:41.974403 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.974369 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\") pod \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " Nov 03 16:52:41.974616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.974423 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\") pod \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " Nov 03 16:52:41.974616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.974497 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\" (UniqueName: \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\") pod \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\" (UID: \"9bc50cff-43a3-4bff-8fb0-b0b648f74427\") " Nov 03 16:52:41.974888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.974861 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca" pod "9bc50cff-43a3-4bff-8fb0-b0b648f74427" (UID: "9bc50cff-43a3-4bff-8fb0-b0b648f74427"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.975182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.975146 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.977853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.977814 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca" pod "9bc50cff-43a3-4bff-8fb0-b0b648f74427" (UID: "9bc50cff-43a3-4bff-8fb0-b0b648f74427"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:41.977988 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:41.977958 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt" pod "9bc50cff-43a3-4bff-8fb0-b0b648f74427" (UID: "9bc50cff-43a3-4bff-8fb0-b0b648f74427"). InnerVolumeSpecName "kube-api-access-ctvbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:42.076683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.076641 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc50cff-43a3-4bff-8fb0-b0b648f74427-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:42.076683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.076675 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctvbt\" (UniqueName: \"kubernetes.io/projected/9bc50cff-43a3-4bff-8fb0-b0b648f74427-kube-api-access-ctvbt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:42.723326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.723290 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" Nov 03 16:52:42.723843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.723320 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm" event={"ID":"9bc50cff-43a3-4bff-8fb0-b0b648f74427","Type":"ContainerDied","Data":"645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63"} Nov 03 16:52:42.723843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.723405 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645857d81fc55c2ebfefb5a6aa678520defe315b030f0dd2c9bfa30aa3996e63" Nov 03 16:52:42.791030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.791000 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" path="/var/lib/kubelet/pods/1125ef8c-f6ce-48fc-9888-b04f7231f30e/volumes" Nov 03 16:52:42.791395 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:42.791381 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204c4df8-bc7d-4237-8643-ad529e1f82a5" path="/var/lib/kubelet/pods/204c4df8-bc7d-4237-8643-ad529e1f82a5/volumes" Nov 03 16:52:45.651151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:45.651114 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:52:45.651602 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:45.651367 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" containerID="cri-o://17133c3d6944424f17f345001606cd4f27839b2af6ff90ecfc94a83ec97b9e4e" gracePeriod=30 Nov 03 16:52:47.122365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:47.122284 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:52:47.122802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:47.122544 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" containerID="cri-o://f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39" gracePeriod=30 Nov 03 16:52:50.851628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:50.851579 6001 generic.go:356] "Generic (PLEG): container finished" podID="04670ce1-1dc0-4373-9db0-707cfa924865" containerID="17133c3d6944424f17f345001606cd4f27839b2af6ff90ecfc94a83ec97b9e4e" exitCode=0 Nov 03 16:52:50.852386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:50.851936 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerDied","Data":"17133c3d6944424f17f345001606cd4f27839b2af6ff90ecfc94a83ec97b9e4e"} Nov 03 16:52:50.852386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:50.852079 6001 scope.go:122] "RemoveContainer" containerID="2ccb91e9c6d42bc7daa809d7f67e86e968e53bde39e790290e1e915f0da19003" Nov 03 16:52:51.005064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.005034 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:52:51.145484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145419 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145508 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145538 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145576 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145604 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145721 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.145899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145886 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.146001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.145934 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:51.146073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146037 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:51.146191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146169 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.146330 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:51.146413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146229 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:51.146413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146362 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\") pod \"04670ce1-1dc0-4373-9db0-707cfa924865\" (UID: \"04670ce1-1dc0-4373-9db0-707cfa924865\") " Nov 03 16:52:51.146607 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146590 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:51.146934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146912 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.147003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146939 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04670ce1-1dc0-4373-9db0-707cfa924865-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.147003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146952 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.147003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146966 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.147003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.146980 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.148233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.148205 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:51.148233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.148227 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:51.148374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.148308 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:51.148374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.148316 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6" pod "04670ce1-1dc0-4373-9db0-707cfa924865" (UID: "04670ce1-1dc0-4373-9db0-707cfa924865"). InnerVolumeSpecName "kube-api-access-mvjs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:51.248693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.248611 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.248693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.248641 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04670ce1-1dc0-4373-9db0-707cfa924865-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.248693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.248651 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04670ce1-1dc0-4373-9db0-707cfa924865-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.248693 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.248665 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvjs6\" (UniqueName: \"kubernetes.io/projected/04670ce1-1dc0-4373-9db0-707cfa924865-kube-api-access-mvjs6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:51.869017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.866914 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" Nov 03 16:52:51.869017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.866946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528" event={"ID":"04670ce1-1dc0-4373-9db0-707cfa924865","Type":"ContainerDied","Data":"d699fc1257b74cb8d19ef75608d686c3d41900037c557380514fdddc7bd56da6"} Nov 03 16:52:51.869017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.867043 6001 scope.go:122] "RemoveContainer" containerID="17133c3d6944424f17f345001606cd4f27839b2af6ff90ecfc94a83ec97b9e4e" Nov 03 16:52:51.946754 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.946721 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:52:51.959671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:51.959638 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-549271-openshift-default-84bcb4d5b5-tx528"] Nov 03 16:52:52.252269 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:52.252235 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:52:52.399434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.399364 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:52:52.465258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465205 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465512 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465557 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465634 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465804 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465787 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465824 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465864 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.465937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465922 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.466003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.465986 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\") pod \"9d883dbb-b29b-47d0-acb8-c61080ac989b\" (UID: \"9d883dbb-b29b-47d0-acb8-c61080ac989b\") " Nov 03 16:52:52.466592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.466534 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:52.466858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.466787 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:52.466858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.466797 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:52.467053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.466883 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:52.467322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467270 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:52.467548 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467522 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.467667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467551 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d883dbb-b29b-47d0-acb8-c61080ac989b-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.467667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467566 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.467667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467583 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.467667 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.467597 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.469232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.468925 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:52.469232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.468958 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:52.469232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.469071 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:52.469739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.469703 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j" pod "9d883dbb-b29b-47d0-acb8-c61080ac989b" (UID: "9d883dbb-b29b-47d0-acb8-c61080ac989b"). InnerVolumeSpecName "kube-api-access-jtb7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:52.569393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.569037 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.569393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.569071 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.569393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.569085 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtb7j\" (UniqueName: \"kubernetes.io/projected/9d883dbb-b29b-47d0-acb8-c61080ac989b-kube-api-access-jtb7j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.569393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.569099 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d883dbb-b29b-47d0-acb8-c61080ac989b-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:52.792993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.792958 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" path="/var/lib/kubelet/pods/04670ce1-1dc0-4373-9db0-707cfa924865/volumes" Nov 03 16:52:52.812652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.812616 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:52:52.813585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813555 6001 cpu_manager.go:414] "containerMap: removing container" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.813585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813584 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813600 6001 cpu_manager.go:414] "containerMap: removing container" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813609 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813621 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813629 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813666 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813673 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813691 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9bc50cff-43a3-4bff-8fb0-b0b648f74427" containerName="revoke-keys" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813698 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc50cff-43a3-4bff-8fb0-b0b648f74427" containerName="revoke-keys" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813710 6001 cpu_manager.go:414] "containerMap: removing container" podUID="204c4df8-bc7d-4237-8643-ad529e1f82a5" containerName="payload-pre-processing" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813717 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204c4df8-bc7d-4237-8643-ad529e1f82a5" containerName="payload-pre-processing" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813753 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" containerName="payload-processing" Nov 03 16:52:52.813760 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813761 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" containerName="payload-processing" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813910 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1125ef8c-f6ce-48fc-9888-b04f7231f30e" containerName="payload-processing" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813931 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813941 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="04670ce1-1dc0-4373-9db0-707cfa924865" containerName="istio-proxy" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813950 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813962 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerName="istio-proxy" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813972 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc50cff-43a3-4bff-8fb0-b0b648f74427" containerName="revoke-keys" Nov 03 16:52:52.814109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.813981 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="204c4df8-bc7d-4237-8643-ad529e1f82a5" containerName="payload-pre-processing" Nov 03 16:52:52.819324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.819301 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.841813 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.841784 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-be8495-openshift-default-dockercfg-pnh84\"" Nov 03 16:52:52.845683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.845653 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871590 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871717 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871746 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871802 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871837 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnks5\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871886 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871919 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871955 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.875535 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.871995 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.880974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.880942 6001 generic.go:356] "Generic (PLEG): container finished" podID="9d883dbb-b29b-47d0-acb8-c61080ac989b" containerID="f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39" exitCode=0 Nov 03 16:52:52.881320 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.881229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerDied","Data":"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39"} Nov 03 16:52:52.881421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.881363 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" event={"ID":"9d883dbb-b29b-47d0-acb8-c61080ac989b","Type":"ContainerDied","Data":"828fff1d1dc9acc5dd17de81bc81a60e15b3b0811c500afd2dfb15500605f5ed"} Nov 03 16:52:52.881421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.881394 6001 scope.go:122] "RemoveContainer" containerID="f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39" Nov 03 16:52:52.881568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.881239 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x" Nov 03 16:52:52.894236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.894215 6001 scope.go:122] "RemoveContainer" containerID="f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69" Nov 03 16:52:52.904437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.904416 6001 scope.go:122] "RemoveContainer" containerID="f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39" Nov 03 16:52:52.904841 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:52.904818 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39\": container with ID starting with f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39 not found: ID does not exist" containerID="f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39" Nov 03 16:52:52.904932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.904857 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39"} err="failed to get container status \"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39\": rpc error: code = NotFound desc = could not find container \"f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39\": container with ID starting with f3e156a128e7c6dd9aa3b37526abd53fcf9a3f0663d36dafdbeeb485da59dc39 not found: ID does not exist" Nov 03 16:52:52.904932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.904883 6001 scope.go:122] "RemoveContainer" containerID="f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69" Nov 03 16:52:52.905161 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:52:52.905143 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69\": container with ID starting with f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69 not found: ID does not exist" containerID="f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69" Nov 03 16:52:52.905214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.905170 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69"} err="failed to get container status \"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69\": rpc error: code = NotFound desc = could not find container \"f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69\": container with ID starting with f8f664c93a78505aa02bd85f2ef12443b556bc6cc8bde183b37bbacb03b11a69 not found: ID does not exist" Nov 03 16:52:52.936654 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.936617 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:52:52.945204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.945164 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f2923b-openshift-default-55b66bd4c7-d7r5x"] Nov 03 16:52:52.973073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973032 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973084 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973148 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973241 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973347 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973417 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973579 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973625 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnks5\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973700 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.973781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973739 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.974030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.973742 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.974030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.974013 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.974431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.974399 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.974572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.974504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.976118 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.976097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:52.976476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:52.976428 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:53.028514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.017767 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnks5\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:53.028514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.020873 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\") pod \"e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:53.133059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.132977 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:53.476999 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.476963 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:52:53.477163 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:53.477053 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7e9ee8_8c83_47b5_b501_d71c503fca13.slice/crio-f5b17537c9ae502c4b1a12a6b9e290e7dc6d0d03819394652ea926a26c6bf62c WatchSource:0}: Error finding container f5b17537c9ae502c4b1a12a6b9e290e7dc6d0d03819394652ea926a26c6bf62c: Status 404 returned error can't find the container with id f5b17537c9ae502c4b1a12a6b9e290e7dc6d0d03819394652ea926a26c6bf62c Nov 03 16:52:53.479687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.479655 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:52:53.479792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.479725 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:52:53.479792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.479754 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:52:53.892868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.892772 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerStarted","Data":"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f"} Nov 03 16:52:53.892868 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.892830 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerStarted","Data":"f5b17537c9ae502c4b1a12a6b9e290e7dc6d0d03819394652ea926a26c6bf62c"} Nov 03 16:52:53.971398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:53.971340 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podStartSLOduration=1.971326159 podStartE2EDuration="1.971326159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:53.969779419 +0000 UTC m=+2269.302992989" watchObservedRunningTime="2026-11-03 16:52:53.971326159 +0000 UTC m=+2269.304539725" Nov 03 16:52:54.134112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.134072 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:54.140208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.140174 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:54.188859 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.188764 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8"] Nov 03 16:52:54.206467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.199645 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.232723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.225766 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:52:54.232723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.225871 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:52:54.232723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.225950 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:52:54.232723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.226049 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:52:54.294476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.270695 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8"] Nov 03 16:52:54.315464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.315415 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.315651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.315502 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwt4c\" (UniqueName: \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.315651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.315599 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.416969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.416931 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.417178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.416994 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwt4c\" (UniqueName: \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.417178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.417059 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.417874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.417842 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.419709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.419684 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.441479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.441392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwt4c\" (UniqueName: \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\") pod \"maas-api-revoke-sub-b27195cc-lj2l8\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.602515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.602476 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:54.727069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.718537 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5"] Nov 03 16:52:54.727069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.725327 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.743655 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.733362 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5"] Nov 03 16:52:54.809770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.809726 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d883dbb-b29b-47d0-acb8-c61080ac989b" path="/var/lib/kubelet/pods/9d883dbb-b29b-47d0-acb8-c61080ac989b/volumes" Nov 03 16:52:54.826786 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.826750 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8t2\" (UniqueName: \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.827016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.826997 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.827304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.827276 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.906771 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.906741 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:54.908752 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.908729 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:52:54.933489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.928664 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8t2\" (UniqueName: \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.933489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.928811 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.933489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.928987 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.933489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.929908 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:54.935113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.935076 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8"] Nov 03 16:52:54.947482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:54.943623 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:55.017525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.015622 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8t2\" (UniqueName: \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\") pod \"maas-api-revoke-sub-22fca59b-x5lr5\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:55.096489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.084080 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:55.460026 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:52:55.459991 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7951fd_5622_4286_a054_47cd5e5e0bdb.slice/crio-1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292 WatchSource:0}: Error finding container 1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292: Status 404 returned error can't find the container with id 1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292 Nov 03 16:52:55.473617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.473565 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5"] Nov 03 16:52:55.914049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.913955 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" event={"ID":"7d7951fd-5622-4286-a054-47cd5e5e0bdb","Type":"ContainerStarted","Data":"1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292"} Nov 03 16:52:55.915797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.915764 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" event={"ID":"44db7791-687e-43ae-95a5-21c828d190a4","Type":"ContainerStarted","Data":"4ad0a41813b9b83de450b27993221bf3b09f1bdfc100d87a605cc3e7fcd086fe"} Nov 03 16:52:55.915968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.915819 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" event={"ID":"44db7791-687e-43ae-95a5-21c828d190a4","Type":"ContainerStarted","Data":"f55540833038af03b9dcf4a8a0552b41aa47e4ee53928789d76c4c2053cabad1"} Nov 03 16:52:55.976611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:55.976563 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" podStartSLOduration=1.976547036 podStartE2EDuration="1.976547036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:55.973956667 +0000 UTC m=+2271.307170236" watchObservedRunningTime="2026-11-03 16:52:55.976547036 +0000 UTC m=+2271.309760605" Nov 03 16:52:56.932601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:56.932491 6001 generic.go:356] "Generic (PLEG): container finished" podID="44db7791-687e-43ae-95a5-21c828d190a4" containerID="4ad0a41813b9b83de450b27993221bf3b09f1bdfc100d87a605cc3e7fcd086fe" exitCode=0 Nov 03 16:52:56.933029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:56.932712 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" event={"ID":"44db7791-687e-43ae-95a5-21c828d190a4","Type":"ContainerDied","Data":"4ad0a41813b9b83de450b27993221bf3b09f1bdfc100d87a605cc3e7fcd086fe"} Nov 03 16:52:56.935917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:56.935889 6001 generic.go:356] "Generic (PLEG): container finished" podID="7d7951fd-5622-4286-a054-47cd5e5e0bdb" containerID="6dda6b1c99c67dea738d290fe9c2a727e1d7bdda17afdc2baf4103ddce2904b4" exitCode=0 Nov 03 16:52:56.936081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:56.935970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" event={"ID":"7d7951fd-5622-4286-a054-47cd5e5e0bdb","Type":"ContainerDied","Data":"6dda6b1c99c67dea738d290fe9c2a727e1d7bdda17afdc2baf4103ddce2904b4"} Nov 03 16:52:58.138781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.138753 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:58.143776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.143754 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:58.179128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179099 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\" (UniqueName: \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\") pod \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " Nov 03 16:52:58.179306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179156 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\") pod \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " Nov 03 16:52:58.179306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179204 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\") pod \"44db7791-687e-43ae-95a5-21c828d190a4\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " Nov 03 16:52:58.179505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\" (UniqueName: \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\") pod \"44db7791-687e-43ae-95a5-21c828d190a4\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " Nov 03 16:52:58.179603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179538 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\") pod \"44db7791-687e-43ae-95a5-21c828d190a4\" (UID: \"44db7791-687e-43ae-95a5-21c828d190a4\") " Nov 03 16:52:58.179603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179490 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca" pod "7d7951fd-5622-4286-a054-47cd5e5e0bdb" (UID: "7d7951fd-5622-4286-a054-47cd5e5e0bdb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:58.179603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179590 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\") pod \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\" (UID: \"7d7951fd-5622-4286-a054-47cd5e5e0bdb\") " Nov 03 16:52:58.179603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.179571 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca" pod "44db7791-687e-43ae-95a5-21c828d190a4" (UID: "44db7791-687e-43ae-95a5-21c828d190a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:58.180232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.180206 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.180373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.180236 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.182647 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.181891 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2" pod "7d7951fd-5622-4286-a054-47cd5e5e0bdb" (UID: "7d7951fd-5622-4286-a054-47cd5e5e0bdb"). InnerVolumeSpecName "kube-api-access-dl8t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:58.182811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.182709 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca" pod "44db7791-687e-43ae-95a5-21c828d190a4" (UID: "44db7791-687e-43ae-95a5-21c828d190a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:58.183122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.183096 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c" pod "44db7791-687e-43ae-95a5-21c828d190a4" (UID: "44db7791-687e-43ae-95a5-21c828d190a4"). InnerVolumeSpecName "kube-api-access-vwt4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:58.183243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.183151 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca" pod "7d7951fd-5622-4286-a054-47cd5e5e0bdb" (UID: "7d7951fd-5622-4286-a054-47cd5e5e0bdb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:58.281674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.281640 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwt4c\" (UniqueName: \"kubernetes.io/projected/44db7791-687e-43ae-95a5-21c828d190a4-kube-api-access-vwt4c\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.281674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.281670 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44db7791-687e-43ae-95a5-21c828d190a4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.281674 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.281680 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d7951fd-5622-4286-a054-47cd5e5e0bdb-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.281922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.281689 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl8t2\" (UniqueName: \"kubernetes.io/projected/7d7951fd-5622-4286-a054-47cd5e5e0bdb-kube-api-access-dl8t2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:52:58.950941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.950907 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" Nov 03 16:52:58.951113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.950939 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8" event={"ID":"44db7791-687e-43ae-95a5-21c828d190a4","Type":"ContainerDied","Data":"f55540833038af03b9dcf4a8a0552b41aa47e4ee53928789d76c4c2053cabad1"} Nov 03 16:52:58.951113 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.950981 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f55540833038af03b9dcf4a8a0552b41aa47e4ee53928789d76c4c2053cabad1" Nov 03 16:52:58.952327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.952306 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" Nov 03 16:52:58.952505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.952338 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5" event={"ID":"7d7951fd-5622-4286-a054-47cd5e5e0bdb","Type":"ContainerDied","Data":"1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292"} Nov 03 16:52:58.952505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:52:58.952371 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1636745d6488c581008124602985d84b5b1c31c28c99d2785e04e5e0af6b3292" Nov 03 16:53:04.500160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500120 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz"] Nov 03 16:53:04.500788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500771 6001 cpu_manager.go:414] "containerMap: removing container" podUID="44db7791-687e-43ae-95a5-21c828d190a4" containerName="revoke-keys" Nov 03 16:53:04.500837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500791 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44db7791-687e-43ae-95a5-21c828d190a4" containerName="revoke-keys" Nov 03 16:53:04.500837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500819 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7d7951fd-5622-4286-a054-47cd5e5e0bdb" containerName="revoke-keys" Nov 03 16:53:04.500837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500824 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7951fd-5622-4286-a054-47cd5e5e0bdb" containerName="revoke-keys" Nov 03 16:53:04.500940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500907 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7951fd-5622-4286-a054-47cd5e5e0bdb" containerName="revoke-keys" Nov 03 16:53:04.500940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.500924 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="44db7791-687e-43ae-95a5-21c828d190a4" containerName="revoke-keys" Nov 03 16:53:04.504358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.504340 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.529784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.529756 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz"] Nov 03 16:53:04.538284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.538253 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:04.538284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.538253 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:04.538537 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.538306 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:04.538726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.538687 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:04.568712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.568672 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r56zq\" (UniqueName: \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.568878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.568833 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.568878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.568876 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.670191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.670149 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.670381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.670210 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.670381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.670336 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r56zq\" (UniqueName: \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.670882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.670861 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.672681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.672662 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.695058 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.695027 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r56zq\" (UniqueName: \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\") pod \"maas-api-revoke-sub-dff11751-bgwfz\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:04.819290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:04.819197 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:05.119372 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:05.119329 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1015cf7_c27c_4918_860f_90d4af34e916.slice/crio-35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957 WatchSource:0}: Error finding container 35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957: Status 404 returned error can't find the container with id 35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957 Nov 03 16:53:05.120000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:05.119975 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz"] Nov 03 16:53:06.010816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:06.010772 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" event={"ID":"f1015cf7-c27c-4918-860f-90d4af34e916","Type":"ContainerStarted","Data":"934b48f6702d38e807d942e0366f58954e045559fe27bbc9caf452024390db19"} Nov 03 16:53:06.011243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:06.010837 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" event={"ID":"f1015cf7-c27c-4918-860f-90d4af34e916","Type":"ContainerStarted","Data":"35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957"} Nov 03 16:53:06.056316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:06.056258 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" podStartSLOduration=2.056240862 podStartE2EDuration="2.056240862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:06.053141086 +0000 UTC m=+2281.386354655" watchObservedRunningTime="2026-11-03 16:53:06.056240862 +0000 UTC m=+2281.389454465" Nov 03 16:53:07.019082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:07.019043 6001 generic.go:356] "Generic (PLEG): container finished" podID="f1015cf7-c27c-4918-860f-90d4af34e916" containerID="934b48f6702d38e807d942e0366f58954e045559fe27bbc9caf452024390db19" exitCode=0 Nov 03 16:53:07.019574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:07.019128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" event={"ID":"f1015cf7-c27c-4918-860f-90d4af34e916","Type":"ContainerDied","Data":"934b48f6702d38e807d942e0366f58954e045559fe27bbc9caf452024390db19"} Nov 03 16:53:08.144186 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.144148 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4"] Nov 03 16:53:08.149378 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.149356 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.165489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.165431 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4"] Nov 03 16:53:08.200650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.200627 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:08.320588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320496 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\") pod \"f1015cf7-c27c-4918-860f-90d4af34e916\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " Nov 03 16:53:08.320753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320607 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\" (UniqueName: \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\") pod \"f1015cf7-c27c-4918-860f-90d4af34e916\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " Nov 03 16:53:08.320753 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320662 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\") pod \"f1015cf7-c27c-4918-860f-90d4af34e916\" (UID: \"f1015cf7-c27c-4918-860f-90d4af34e916\") " Nov 03 16:53:08.320885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320811 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca" pod "f1015cf7-c27c-4918-860f-90d4af34e916" (UID: "f1015cf7-c27c-4918-860f-90d4af34e916"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:08.320885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320859 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvpr6\" (UniqueName: \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.320994 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.320924 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.321166 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.321141 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.321344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.321331 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:08.322722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.322702 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca" pod "f1015cf7-c27c-4918-860f-90d4af34e916" (UID: "f1015cf7-c27c-4918-860f-90d4af34e916"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:08.322843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.322825 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq" pod "f1015cf7-c27c-4918-860f-90d4af34e916" (UID: "f1015cf7-c27c-4918-860f-90d4af34e916"). InnerVolumeSpecName "kube-api-access-r56zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:08.422810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.422768 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.422980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.422896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvpr6\" (UniqueName: \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.422980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.422931 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.423048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.423000 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r56zq\" (UniqueName: \"kubernetes.io/projected/f1015cf7-c27c-4918-860f-90d4af34e916-kube-api-access-r56zq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:08.423048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.423011 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1015cf7-c27c-4918-860f-90d4af34e916-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:08.423434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.423404 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.425488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.425465 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.447289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.447255 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvpr6\" (UniqueName: \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\") pod \"maas-api-revoke-sub-5b76ef2b-kv7v4\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.464257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.464212 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:08.774623 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:08.774593 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2098b23_6f88_4ed0_97f2_f69170e4b8ac.slice/crio-e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61 WatchSource:0}: Error finding container e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61: Status 404 returned error can't find the container with id e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61 Nov 03 16:53:08.774758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:08.774717 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4"] Nov 03 16:53:09.035180 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:09.035097 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" event={"ID":"c2098b23-6f88-4ed0-97f2-f69170e4b8ac","Type":"ContainerStarted","Data":"e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61"} Nov 03 16:53:09.036603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:09.036582 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" Nov 03 16:53:09.036603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:09.036597 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz" event={"ID":"f1015cf7-c27c-4918-860f-90d4af34e916","Type":"ContainerDied","Data":"35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957"} Nov 03 16:53:09.036766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:09.036632 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35ccbb825986e5c82f6477f4176d428846d5af58581380de6927db0e2c6d2957" Nov 03 16:53:10.044911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:10.044872 6001 generic.go:356] "Generic (PLEG): container finished" podID="c2098b23-6f88-4ed0-97f2-f69170e4b8ac" containerID="32b977b1b355ef57adcbd9ff5a4a4dd3e48e8f6fe7e6933eb78a5c0ff030997a" exitCode=0 Nov 03 16:53:10.045309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:10.044957 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" event={"ID":"c2098b23-6f88-4ed0-97f2-f69170e4b8ac","Type":"ContainerDied","Data":"32b977b1b355ef57adcbd9ff5a4a4dd3e48e8f6fe7e6933eb78a5c0ff030997a"} Nov 03 16:53:11.214309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.214278 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:11.268944 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.268910 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\") pod \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " Nov 03 16:53:11.269134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.269007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\") pod \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " Nov 03 16:53:11.269134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.269099 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\" (UniqueName: \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\") pod \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\" (UID: \"c2098b23-6f88-4ed0-97f2-f69170e4b8ac\") " Nov 03 16:53:11.269303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.269276 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca" pod "c2098b23-6f88-4ed0-97f2-f69170e4b8ac" (UID: "c2098b23-6f88-4ed0-97f2-f69170e4b8ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:11.269672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.269633 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.271221 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.271188 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca" pod "c2098b23-6f88-4ed0-97f2-f69170e4b8ac" (UID: "c2098b23-6f88-4ed0-97f2-f69170e4b8ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:11.271385 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.271366 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6" pod "c2098b23-6f88-4ed0-97f2-f69170e4b8ac" (UID: "c2098b23-6f88-4ed0-97f2-f69170e4b8ac"). InnerVolumeSpecName "kube-api-access-bvpr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:11.371387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.371302 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.371387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.371334 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvpr6\" (UniqueName: \"kubernetes.io/projected/c2098b23-6f88-4ed0-97f2-f69170e4b8ac-kube-api-access-bvpr6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:11.637498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.637403 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd"] Nov 03 16:53:11.638009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.637996 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c2098b23-6f88-4ed0-97f2-f69170e4b8ac" containerName="revoke-keys" Nov 03 16:53:11.638072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.638011 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2098b23-6f88-4ed0-97f2-f69170e4b8ac" containerName="revoke-keys" Nov 03 16:53:11.638072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.638029 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f1015cf7-c27c-4918-860f-90d4af34e916" containerName="revoke-keys" Nov 03 16:53:11.638072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.638035 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1015cf7-c27c-4918-860f-90d4af34e916" containerName="revoke-keys" Nov 03 16:53:11.638332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.638128 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2098b23-6f88-4ed0-97f2-f69170e4b8ac" containerName="revoke-keys" Nov 03 16:53:11.638332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.638137 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1015cf7-c27c-4918-860f-90d4af34e916" containerName="revoke-keys" Nov 03 16:53:11.641382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.641365 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.661461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.661413 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd"] Nov 03 16:53:11.674518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.674482 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.674768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.674748 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nthx8\" (UniqueName: \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.674878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.674862 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.776097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.776060 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.776304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.776283 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nthx8\" (UniqueName: \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.776401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.776385 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.777238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.777207 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.779051 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.779014 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.801419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.801390 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nthx8\" (UniqueName: \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\") pod \"maas-api-revoke-sub-f1f9a7cb-vcsvd\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:11.956031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:11.955940 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:12.059861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:12.059835 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" Nov 03 16:53:12.060044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:12.059866 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4" event={"ID":"c2098b23-6f88-4ed0-97f2-f69170e4b8ac","Type":"ContainerDied","Data":"e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61"} Nov 03 16:53:12.060044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:12.059910 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6d0c38667b8713d572832217ca01bc8fc5d4572cc69d1d1ad68de2b9b019d61" Nov 03 16:53:12.258424 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:12.258390 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf29677a_31e0_47ee_bbd5_923bad224384.slice/crio-3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f WatchSource:0}: Error finding container 3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f: Status 404 returned error can't find the container with id 3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f Nov 03 16:53:12.259383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:12.259359 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd"] Nov 03 16:53:13.069064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:13.069025 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" event={"ID":"df29677a-31e0-47ee-bbd5-923bad224384","Type":"ContainerStarted","Data":"b5b0dd921d4c9d6e764485ebb5f0bd6aa4b2e883851a870b1a1ec77861d086dd"} Nov 03 16:53:13.069233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:13.069109 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" event={"ID":"df29677a-31e0-47ee-bbd5-923bad224384","Type":"ContainerStarted","Data":"3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f"} Nov 03 16:53:13.113183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:13.113124 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" podStartSLOduration=2.113109867 podStartE2EDuration="2.113109867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:13.111480599 +0000 UTC m=+2288.444694165" watchObservedRunningTime="2026-11-03 16:53:13.113109867 +0000 UTC m=+2288.446323436" Nov 03 16:53:14.078496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:14.078419 6001 generic.go:356] "Generic (PLEG): container finished" podID="df29677a-31e0-47ee-bbd5-923bad224384" containerID="b5b0dd921d4c9d6e764485ebb5f0bd6aa4b2e883851a870b1a1ec77861d086dd" exitCode=0 Nov 03 16:53:14.078901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:14.078509 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" event={"ID":"df29677a-31e0-47ee-bbd5-923bad224384","Type":"ContainerDied","Data":"b5b0dd921d4c9d6e764485ebb5f0bd6aa4b2e883851a870b1a1ec77861d086dd"} Nov 03 16:53:15.368927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.368895 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:15.541006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.540957 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\") pod \"df29677a-31e0-47ee-bbd5-923bad224384\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " Nov 03 16:53:15.541203 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.541151 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\" (UniqueName: \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\") pod \"df29677a-31e0-47ee-bbd5-923bad224384\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " Nov 03 16:53:15.541313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.541289 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\") pod \"df29677a-31e0-47ee-bbd5-923bad224384\" (UID: \"df29677a-31e0-47ee-bbd5-923bad224384\") " Nov 03 16:53:15.541519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.541435 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca" pod "df29677a-31e0-47ee-bbd5-923bad224384" (UID: "df29677a-31e0-47ee-bbd5-923bad224384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:15.541932 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.541900 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:15.544338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.544290 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca" pod "df29677a-31e0-47ee-bbd5-923bad224384" (UID: "df29677a-31e0-47ee-bbd5-923bad224384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:15.544712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.544682 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8" pod "df29677a-31e0-47ee-bbd5-923bad224384" (UID: "df29677a-31e0-47ee-bbd5-923bad224384"). InnerVolumeSpecName "kube-api-access-nthx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:15.643620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.643499 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nthx8\" (UniqueName: \"kubernetes.io/projected/df29677a-31e0-47ee-bbd5-923bad224384-kube-api-access-nthx8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:15.643620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:15.643560 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df29677a-31e0-47ee-bbd5-923bad224384-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:16.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:16.127951 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" event={"ID":"df29677a-31e0-47ee-bbd5-923bad224384","Type":"ContainerDied","Data":"3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f"} Nov 03 16:53:16.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:16.128019 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2945f207b39193b23a7c769e0a4b5c16c79af92e0bcde983f53c0665ed7a6f" Nov 03 16:53:16.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:16.128091 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd" Nov 03 16:53:19.712619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.712583 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 16:53:19.713553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.713532 6001 cpu_manager.go:414] "containerMap: removing container" podUID="df29677a-31e0-47ee-bbd5-923bad224384" containerName="revoke-keys" Nov 03 16:53:19.713619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.713556 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df29677a-31e0-47ee-bbd5-923bad224384" containerName="revoke-keys" Nov 03 16:53:19.713719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.713707 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="df29677a-31e0-47ee-bbd5-923bad224384" containerName="revoke-keys" Nov 03 16:53:19.718063 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.718041 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.738365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.738336 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 16:53:19.741653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.741629 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.741942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.741921 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.741983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742027 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742092 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742191 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742256 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.742408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.742283 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tjm4\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843404 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843369 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843578 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843659 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843696 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843906 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843870 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843953 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.843992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.843969 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844060 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844149 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844222 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844179 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tjm4\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844255 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844333 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.844439 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.844418 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.846241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.846221 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.846673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.846653 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.867757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.867729 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:19.867866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:19.867804 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tjm4\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:20.043707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.043668 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:20.380134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.380100 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 16:53:20.380322 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:20.380291 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d011fb_b13f_447e_962a_4601bb81eadb.slice/crio-170220a44b8223b825d96dec1c139fd7e2d6b596f03daa19cfae72acbd397785 WatchSource:0}: Error finding container 170220a44b8223b825d96dec1c139fd7e2d6b596f03daa19cfae72acbd397785: Status 404 returned error can't find the container with id 170220a44b8223b825d96dec1c139fd7e2d6b596f03daa19cfae72acbd397785 Nov 03 16:53:20.382797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.382765 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:20.382876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.382831 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:20.382876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.382859 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:20.465679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.455655 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4"] Nov 03 16:53:20.476731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.476696 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4"] Nov 03 16:53:20.477133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.477065 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.495534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.495509 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:20.495653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.495598 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:20.495653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.495640 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:20.495765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.495695 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:20.553714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.553683 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvtx\" (UniqueName: \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.553854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.553837 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.553921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.553902 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.655254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.655168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvtx\" (UniqueName: \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.655414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.655282 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.655414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.655335 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.656265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.656235 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.657853 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.657828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.692885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.692787 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvtx\" (UniqueName: \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\") pod \"maas-api-revoke-sub-068f5bc3-g22m4\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:20.799379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:20.799346 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:21.138630 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:21.138596 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c387f6d_864c_44c9_8ab4_a63e0c112bb2.slice/crio-fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a WatchSource:0}: Error finding container fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a: Status 404 returned error can't find the container with id fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a Nov 03 16:53:21.139286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:21.139256 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4"] Nov 03 16:53:21.165479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:21.165434 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" event={"ID":"8c387f6d-864c-44c9-8ab4-a63e0c112bb2","Type":"ContainerStarted","Data":"fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a"} Nov 03 16:53:21.167586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:21.167554 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" event={"ID":"76d011fb-b13f-447e-962a-4601bb81eadb","Type":"ContainerStarted","Data":"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe"} Nov 03 16:53:21.167738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:21.167602 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" event={"ID":"76d011fb-b13f-447e-962a-4601bb81eadb","Type":"ContainerStarted","Data":"170220a44b8223b825d96dec1c139fd7e2d6b596f03daa19cfae72acbd397785"} Nov 03 16:53:21.229951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:21.229896 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podStartSLOduration=2.229878497 podStartE2EDuration="2.229878497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:21.229175055 +0000 UTC m=+2296.562388624" watchObservedRunningTime="2026-11-03 16:53:21.229878497 +0000 UTC m=+2296.563092066" Nov 03 16:53:22.050851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:22.050813 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:22.068697 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:22.068646 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:22.068937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:22.068743 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:22.177427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:22.177387 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" event={"ID":"8c387f6d-864c-44c9-8ab4-a63e0c112bb2","Type":"ContainerStarted","Data":"d952af9d128aff326975c1cc8efff2a760d8c8e72f4dfbe5031de0b6b7accf56"} Nov 03 16:53:22.221191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:22.221138 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" podStartSLOduration=2.221124774 podStartE2EDuration="2.221124774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:22.219368023 +0000 UTC m=+2297.552581593" watchObservedRunningTime="2026-11-03 16:53:22.221124774 +0000 UTC m=+2297.554338344" Nov 03 16:53:22.278523 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:53:22.278457 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:53:23.067467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:23.066880 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:23.067467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:23.066957 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:23.185662 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:23.185623 6001 generic.go:356] "Generic (PLEG): container finished" podID="8c387f6d-864c-44c9-8ab4-a63e0c112bb2" containerID="d952af9d128aff326975c1cc8efff2a760d8c8e72f4dfbe5031de0b6b7accf56" exitCode=0 Nov 03 16:53:23.185824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:23.185705 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" event={"ID":"8c387f6d-864c-44c9-8ab4-a63e0c112bb2","Type":"ContainerDied","Data":"d952af9d128aff326975c1cc8efff2a760d8c8e72f4dfbe5031de0b6b7accf56"} Nov 03 16:53:24.055477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.054994 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.055477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.055054 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:24.398908 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.398882 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:24.410029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.409999 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\") pod \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " Nov 03 16:53:24.410224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.410209 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\" (UniqueName: \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\") pod \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " Nov 03 16:53:24.410297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.410253 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\") pod \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\" (UID: \"8c387f6d-864c-44c9-8ab4-a63e0c112bb2\") " Nov 03 16:53:24.410369 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.410294 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca" pod "8c387f6d-864c-44c9-8ab4-a63e0c112bb2" (UID: "8c387f6d-864c-44c9-8ab4-a63e0c112bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:24.410700 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.410663 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:24.412312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.412285 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca" pod "8c387f6d-864c-44c9-8ab4-a63e0c112bb2" (UID: "8c387f6d-864c-44c9-8ab4-a63e0c112bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:24.412397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.412378 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx" pod "8c387f6d-864c-44c9-8ab4-a63e0c112bb2" (UID: "8c387f6d-864c-44c9-8ab4-a63e0c112bb2"). InnerVolumeSpecName "kube-api-access-tgvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:24.512078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.512039 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgvtx\" (UniqueName: \"kubernetes.io/projected/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-kube-api-access-tgvtx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:24.512078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:24.512072 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c387f6d-864c-44c9-8ab4-a63e0c112bb2-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:25.056021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.055917 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:25.056303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.056272 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:25.199847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.199817 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" Nov 03 16:53:25.200026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.199816 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4" event={"ID":"8c387f6d-864c-44c9-8ab4-a63e0c112bb2","Type":"ContainerDied","Data":"fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a"} Nov 03 16:53:25.200026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.199938 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2c522edfee9ef9a27fe6327c45002d8d62f259dd109456b5fe34a802ee0a3a" Nov 03 16:53:25.587334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.587303 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf"] Nov 03 16:53:25.588025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.588005 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8c387f6d-864c-44c9-8ab4-a63e0c112bb2" containerName="revoke-keys" Nov 03 16:53:25.588025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.588025 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c387f6d-864c-44c9-8ab4-a63e0c112bb2" containerName="revoke-keys" Nov 03 16:53:25.588153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.588137 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c387f6d-864c-44c9-8ab4-a63e0c112bb2" containerName="revoke-keys" Nov 03 16:53:25.591930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.591910 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.620802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.620772 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:25.621183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.621157 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:25.621309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.621234 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:25.621375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.621311 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:25.628226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.628181 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf"] Nov 03 16:53:25.632967 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.632926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwlwl\" (UniqueName: \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.633191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.633058 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.633191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.633132 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.734923 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.734874 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwlwl\" (UniqueName: \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.735131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.734982 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.735131 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.735051 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.735663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.735640 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.737481 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.737439 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.773489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.771178 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwlwl\" (UniqueName: \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\") pod \"maas-api-revoke-sub-90950915-7qxwf\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:25.907625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:25.907525 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:26.067469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:26.051934 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:26.067469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:26.052045 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:26.234948 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:26.234913 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72e0a0c_3f3f_4e5f_a068_8aa8ba9dca30.slice/crio-d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3 WatchSource:0}: Error finding container d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3: Status 404 returned error can't find the container with id d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3 Nov 03 16:53:26.235800 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:26.235772 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf"] Nov 03 16:53:27.044836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:27.044788 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.045495 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:27.044867 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" Nov 03 16:53:27.216627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:27.216591 6001 generic.go:356] "Generic (PLEG): container finished" podID="e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" containerID="f486cb759d473d119a6fb7dccdec17685076fb5ea6332342ebb20650baa24eb5" exitCode=0 Nov 03 16:53:27.216826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:27.216667 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" event={"ID":"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30","Type":"ContainerDied","Data":"f486cb759d473d119a6fb7dccdec17685076fb5ea6332342ebb20650baa24eb5"} Nov 03 16:53:27.216826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:27.216721 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" event={"ID":"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30","Type":"ContainerStarted","Data":"d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3"} Nov 03 16:53:28.340220 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.340183 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:28.341704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.341680 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:28.342874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.342851 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 16:53:28.394719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.394695 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:28.491228 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.491123 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\") pod \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " Nov 03 16:53:28.491625 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.491592 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\") pod \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " Nov 03 16:53:28.491766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.491748 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\" (UniqueName: \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\") pod \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\" (UID: \"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30\") " Nov 03 16:53:28.495414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.495380 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca" pod "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" (UID: "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:28.498359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.498329 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl" pod "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" (UID: "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30"). InnerVolumeSpecName "kube-api-access-mwlwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:28.498496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.498375 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca" pod "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" (UID: "e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:28.593836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.593796 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:28.593836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.593832 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:28.593836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:28.593843 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwlwl\" (UniqueName: \"kubernetes.io/projected/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30-kube-api-access-mwlwl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:29.231230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:29.231204 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" Nov 03 16:53:29.231435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:29.231243 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf" event={"ID":"e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30","Type":"ContainerDied","Data":"d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3"} Nov 03 16:53:29.231435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:29.231302 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d39fc9130e657562e04fd313874d15a59ca7af2988b8d597ed5456b31517dab3" Nov 03 16:53:31.516331 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.516281 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:53:31.517852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.517823 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" containerName="revoke-keys" Nov 03 16:53:31.518041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.518017 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" containerName="revoke-keys" Nov 03 16:53:31.518273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.518258 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" containerName="revoke-keys" Nov 03 16:53:31.522436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.522404 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.563737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.563701 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a11d94-dockercfg-vjm42\"" Nov 03 16:53:31.565684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.565554 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a11d94\"" Nov 03 16:53:31.582001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.581952 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:53:31.646833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.646799 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.647006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.646960 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llstr\" (UniqueName: \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.747862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.747824 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.748045 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.747938 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llstr\" (UniqueName: \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.748686 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.748664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.773316 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.773241 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llstr\" (UniqueName: \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\") pod \"payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:31.853553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:31.853510 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:32.163839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:32.163738 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:53:32.165954 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:32.165916 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c12a8e8_3084_467a_b8d7_dac939d1447d.slice/crio-57be76651e9b3f15ea19bcd8515d840b032e488e4011463f376c6409e242ab53 WatchSource:0}: Error finding container 57be76651e9b3f15ea19bcd8515d840b032e488e4011463f376c6409e242ab53: Status 404 returned error can't find the container with id 57be76651e9b3f15ea19bcd8515d840b032e488e4011463f376c6409e242ab53 Nov 03 16:53:32.267021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:32.266979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" event={"ID":"6c12a8e8-3084-467a-b8d7-dac939d1447d","Type":"ContainerStarted","Data":"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091"} Nov 03 16:53:32.267219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:32.267038 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" event={"ID":"6c12a8e8-3084-467a-b8d7-dac939d1447d","Type":"ContainerStarted","Data":"57be76651e9b3f15ea19bcd8515d840b032e488e4011463f376c6409e242ab53"} Nov 03 16:53:32.267219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:32.267064 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:32.310343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:32.310285 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" podStartSLOduration=1.310269352 podStartE2EDuration="1.310269352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:32.309479703 +0000 UTC m=+2307.642693273" watchObservedRunningTime="2026-11-03 16:53:32.310269352 +0000 UTC m=+2307.643482921" Nov 03 16:53:34.849103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.849064 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9"] Nov 03 16:53:34.853459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.853430 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:34.873626 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.873599 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9"] Nov 03 16:53:34.882321 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.882297 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:34.882469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.882287 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:34.882581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.882349 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:34.882722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.882664 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:34.991747 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.991713 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:34.991992 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.991966 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:34.992103 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:34.992088 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47wwj\" (UniqueName: \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.093768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.093732 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.093951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.093808 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47wwj\" (UniqueName: \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.093951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.093866 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.094488 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.094437 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.096087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.096063 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.118012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.117943 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47wwj\" (UniqueName: \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\") pod \"maas-api-revoke-sub-887accd4-2t5p9\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.167050 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.167015 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:35.468981 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:35.468949 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403af292_5eea_4b4b_bd5a_5b496f5af605.slice/crio-15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da WatchSource:0}: Error finding container 15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da: Status 404 returned error can't find the container with id 15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da Nov 03 16:53:35.471064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:35.471036 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9"] Nov 03 16:53:36.303231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:36.303181 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" event={"ID":"403af292-5eea-4b4b-bd5a-5b496f5af605","Type":"ContainerStarted","Data":"74942f4f01e1cef36b567717875b95196008aea98786e9ad8265f30064b8332d"} Nov 03 16:53:36.303726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:36.303240 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" event={"ID":"403af292-5eea-4b4b-bd5a-5b496f5af605","Type":"ContainerStarted","Data":"15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da"} Nov 03 16:53:36.347574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:36.347510 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" podStartSLOduration=2.347498107 podStartE2EDuration="2.347498107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:36.345767286 +0000 UTC m=+2311.678980857" watchObservedRunningTime="2026-11-03 16:53:36.347498107 +0000 UTC m=+2311.680711675" Nov 03 16:53:37.351334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:37.351294 6001 generic.go:356] "Generic (PLEG): container finished" podID="403af292-5eea-4b4b-bd5a-5b496f5af605" containerID="74942f4f01e1cef36b567717875b95196008aea98786e9ad8265f30064b8332d" exitCode=0 Nov 03 16:53:37.353734 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:37.353702 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" event={"ID":"403af292-5eea-4b4b-bd5a-5b496f5af605","Type":"ContainerDied","Data":"74942f4f01e1cef36b567717875b95196008aea98786e9ad8265f30064b8332d"} Nov 03 16:53:38.289577 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.289529 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.455634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.455576 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.799465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.795542 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.799465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.796758 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.799465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.797428 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.799465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.797701 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 16:53:38.918473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:38.891541 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 16:53:39.034619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.034541 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.278466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.132506 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 16:53:39.405539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.405513 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 16:53:39.405842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.405829 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.406219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.406160 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:39.516696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.516670 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" Nov 03 16:53:39.517323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.516769 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9" event={"ID":"403af292-5eea-4b4b-bd5a-5b496f5af605","Type":"ContainerDied","Data":"15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da"} Nov 03 16:53:39.517520 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.517503 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15db8be7ff60cca9e712d0cd77be8a3b474cd4873b408955c7f28953de83f8da" Nov 03 16:53:39.581088 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.580994 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\") pod \"403af292-5eea-4b4b-bd5a-5b496f5af605\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " Nov 03 16:53:39.581246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581212 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\" (UniqueName: \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\") pod \"403af292-5eea-4b4b-bd5a-5b496f5af605\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " Nov 03 16:53:39.581246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581233 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\") pod \"403af292-5eea-4b4b-bd5a-5b496f5af605\" (UID: \"403af292-5eea-4b4b-bd5a-5b496f5af605\") " Nov 03 16:53:39.581530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581499 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.581530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581545 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.581795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581687 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca" pod "403af292-5eea-4b4b-bd5a-5b496f5af605" (UID: "403af292-5eea-4b4b-bd5a-5b496f5af605"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:39.581795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581711 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.581795 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581769 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.581974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581822 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.581974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581892 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.582095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.581984 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzkv\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.582095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.582006 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.582209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.582096 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.582269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.582254 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:39.583344 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.583321 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca" pod "403af292-5eea-4b4b-bd5a-5b496f5af605" (UID: "403af292-5eea-4b4b-bd5a-5b496f5af605"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:39.583511 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.583377 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj" pod "403af292-5eea-4b4b-bd5a-5b496f5af605" (UID: "403af292-5eea-4b4b-bd5a-5b496f5af605"). InnerVolumeSpecName "kube-api-access-47wwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:39.683554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683672 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzkv\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683707 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683733 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683778 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683808 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.683945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683912 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683946 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.683979 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684102 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684053 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684133 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47wwj\" (UniqueName: \"kubernetes.io/projected/403af292-5eea-4b4b-bd5a-5b496f5af605-kube-api-access-47wwj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:39.684261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684152 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/403af292-5eea-4b4b-bd5a-5b496f5af605-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:39.684492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684388 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684433 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684474 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.684943 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.684921 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.686357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.686340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.686780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.686763 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.707914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.707883 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.708062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.708023 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzkv\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:39.726690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:39.726661 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:40.023661 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.023626 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9"] Nov 03 16:53:40.024242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.024228 6001 cpu_manager.go:414] "containerMap: removing container" podUID="403af292-5eea-4b4b-bd5a-5b496f5af605" containerName="revoke-keys" Nov 03 16:53:40.024294 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.024244 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403af292-5eea-4b4b-bd5a-5b496f5af605" containerName="revoke-keys" Nov 03 16:53:40.024356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.024347 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="403af292-5eea-4b4b-bd5a-5b496f5af605" containerName="revoke-keys" Nov 03 16:53:40.029064 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.029044 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.063468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.062876 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:40.063468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.062976 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:40.063468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.063121 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:40.063815 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.063763 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:40.078342 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:40.078310 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964b9fa7_e304_4c95_aa7b_c152c097da64.slice/crio-68da4eb3dd5be1ea465fb38b6578c5740ec156582c7b0a953122fa2b478bd595 WatchSource:0}: Error finding container 68da4eb3dd5be1ea465fb38b6578c5740ec156582c7b0a953122fa2b478bd595: Status 404 returned error can't find the container with id 68da4eb3dd5be1ea465fb38b6578c5740ec156582c7b0a953122fa2b478bd595 Nov 03 16:53:40.078342 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.078319 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9"] Nov 03 16:53:40.081299 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.081261 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:40.081387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.081353 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:40.081424 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.081413 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:40.081887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.081858 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 16:53:40.089219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.089190 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.089375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.089351 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.089529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.089504 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7z9\" (UniqueName: \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.191072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.191032 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.191281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.191167 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7z9\" (UniqueName: \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.191281 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.191268 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.191742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.191717 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.193723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.193703 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.215128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.215090 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7z9\" (UniqueName: \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\") pod \"maas-api-revoke-sub-cb42f7c0-76qm9\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.343499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.343392 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:40.526313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.526272 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" event={"ID":"964b9fa7-e304-4c95-aa7b-c152c097da64","Type":"ContainerStarted","Data":"d5e997cec03b80308e5189ab86bcc2af805193ce1bc8a3566e407503437737cd"} Nov 03 16:53:40.526756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.526335 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" event={"ID":"964b9fa7-e304-4c95-aa7b-c152c097da64","Type":"ContainerStarted","Data":"68da4eb3dd5be1ea465fb38b6578c5740ec156582c7b0a953122fa2b478bd595"} Nov 03 16:53:40.575593 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.575538 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podStartSLOduration=2.57552193 podStartE2EDuration="2.57552193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:40.573348991 +0000 UTC m=+2315.906562594" watchObservedRunningTime="2026-11-03 16:53:40.57552193 +0000 UTC m=+2315.908735498" Nov 03 16:53:40.671839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.671718 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9"] Nov 03 16:53:40.674386 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:40.674344 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f7a226_f383_49e7_9c5f_dfa0eea71933.slice/crio-84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c WatchSource:0}: Error finding container 84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c: Status 404 returned error can't find the container with id 84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c Nov 03 16:53:40.727322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:40.727297 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:41.536838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:41.536799 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" event={"ID":"57f7a226-f383-49e7-9c5f-dfa0eea71933","Type":"ContainerStarted","Data":"caee76d9f3c2283bf79aa0ff607929c65fce37baf4930adb608476986d329e53"} Nov 03 16:53:41.537245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:41.536858 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" event={"ID":"57f7a226-f383-49e7-9c5f-dfa0eea71933","Type":"ContainerStarted","Data":"84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c"} Nov 03 16:53:41.729167 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:41.729132 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:41.729336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:41.729187 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:42.591470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:42.584533 6001 generic.go:356] "Generic (PLEG): container finished" podID="57f7a226-f383-49e7-9c5f-dfa0eea71933" containerID="caee76d9f3c2283bf79aa0ff607929c65fce37baf4930adb608476986d329e53" exitCode=0 Nov 03 16:53:42.591470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:42.585960 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" event={"ID":"57f7a226-f383-49e7-9c5f-dfa0eea71933","Type":"ContainerDied","Data":"caee76d9f3c2283bf79aa0ff607929c65fce37baf4930adb608476986d329e53"} Nov 03 16:53:42.727067 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:42.727024 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:42.727390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:42.727367 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:43.290636 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.290598 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:53:43.727327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.727289 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:43.727834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.727351 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:43.832921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.832885 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:43.876172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.876078 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\" (UniqueName: \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\") pod \"57f7a226-f383-49e7-9c5f-dfa0eea71933\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " Nov 03 16:53:43.876323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.876187 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\") pod \"57f7a226-f383-49e7-9c5f-dfa0eea71933\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " Nov 03 16:53:43.876323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.876226 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\") pod \"57f7a226-f383-49e7-9c5f-dfa0eea71933\" (UID: \"57f7a226-f383-49e7-9c5f-dfa0eea71933\") " Nov 03 16:53:43.876547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.876523 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca" pod "57f7a226-f383-49e7-9c5f-dfa0eea71933" (UID: "57f7a226-f383-49e7-9c5f-dfa0eea71933"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:43.876986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.876960 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:43.878598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.878573 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9" pod "57f7a226-f383-49e7-9c5f-dfa0eea71933" (UID: "57f7a226-f383-49e7-9c5f-dfa0eea71933"). InnerVolumeSpecName "kube-api-access-qn7z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:43.878707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.878617 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca" pod "57f7a226-f383-49e7-9c5f-dfa0eea71933" (UID: "57f7a226-f383-49e7-9c5f-dfa0eea71933"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:43.978379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.978340 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn7z9\" (UniqueName: \"kubernetes.io/projected/57f7a226-f383-49e7-9c5f-dfa0eea71933-kube-api-access-qn7z9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:43.978379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:43.978381 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57f7a226-f383-49e7-9c5f-dfa0eea71933-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:44.606917 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:44.606880 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" event={"ID":"57f7a226-f383-49e7-9c5f-dfa0eea71933","Type":"ContainerDied","Data":"84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c"} Nov 03 16:53:44.607081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:44.606923 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9" Nov 03 16:53:44.607081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:44.606933 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84aa75da1c005ce639dcf54b6d82029f1855a4cef0da312a3056a402b2cbc18c" Nov 03 16:53:44.727046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:44.727001 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:44.727239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:44.727075 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:45.727120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:45.727080 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:45.727738 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:45.727699 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:46.567434 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.567397 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5"] Nov 03 16:53:46.568094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.568076 6001 cpu_manager.go:414] "containerMap: removing container" podUID="57f7a226-f383-49e7-9c5f-dfa0eea71933" containerName="revoke-keys" Nov 03 16:53:46.568147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.568096 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f7a226-f383-49e7-9c5f-dfa0eea71933" containerName="revoke-keys" Nov 03 16:53:46.568208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.568200 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f7a226-f383-49e7-9c5f-dfa0eea71933" containerName="revoke-keys" Nov 03 16:53:46.572643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.572620 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.591426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.591395 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5"] Nov 03 16:53:46.600340 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.600311 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:46.600660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.600424 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:46.600787 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.600539 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:46.600847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.600614 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:46.635074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.635032 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.635074 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.635079 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m56l\" (UniqueName: \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.635355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.635198 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.727115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.727074 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Nov 03 16:53:46.727290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.727138 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Nov 03 16:53:46.736561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.736523 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.736748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.736581 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m56l\" (UniqueName: \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.736748 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.736634 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.737302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.737272 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.739029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.739004 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.761312 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.761276 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m56l\" (UniqueName: \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\") pod \"maas-api-revoke-sub-9166371b-95mp5\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:46.890407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:46.890307 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:47.234006 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:47.233977 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaf0059_e09c_4b10_9a00_4971b7480aa0.slice/crio-1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1 WatchSource:0}: Error finding container 1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1: Status 404 returned error can't find the container with id 1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1 Nov 03 16:53:47.235157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:47.235124 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5"] Nov 03 16:53:47.633712 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:47.633670 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" event={"ID":"acaf0059-e09c-4b10-9a00-4971b7480aa0","Type":"ContainerStarted","Data":"1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1"} Nov 03 16:53:47.984966 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:47.984925 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:47.985477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:47.985316 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:47.986256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:47.986230 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 16:53:48.642019 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.641939 6001 generic.go:356] "Generic (PLEG): container finished" podID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerID="5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f" exitCode=0 Nov 03 16:53:48.642217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.642016 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerDied","Data":"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f"} Nov 03 16:53:48.642525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.642502 6001 scope.go:122] "RemoveContainer" containerID="5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f" Nov 03 16:53:48.643862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.643840 6001 generic.go:356] "Generic (PLEG): container finished" podID="acaf0059-e09c-4b10-9a00-4971b7480aa0" containerID="78373327e0b22ad9d547ce868ab34ba00a3b8f499f9cf01145995d50a4789517" exitCode=0 Nov 03 16:53:48.643957 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.643920 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" event={"ID":"acaf0059-e09c-4b10-9a00-4971b7480aa0","Type":"ContainerDied","Data":"78373327e0b22ad9d547ce868ab34ba00a3b8f499f9cf01145995d50a4789517"} Nov 03 16:53:48.644172 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.644149 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:48.644223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.644211 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:48.644261 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:48.644245 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:49.131470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.128320 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 16:53:49.135464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.133310 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.245499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.199620 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291293 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291408 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291435 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291486 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291528 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291756 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291820 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthnx\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291855 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.293542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.291896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.392947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.392844 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.392947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.392934 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.392954 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.392976 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393136 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393296 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393399 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393371 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthnx\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393405 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393413 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393423 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393501 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.393750 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393729 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.394023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.393997 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.394232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.394212 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.395676 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.395655 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.395844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.395821 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.418360 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.418334 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.418528 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.418481 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthnx\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.596376 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.596341 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:49.678842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.677521 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerStarted","Data":"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3"} Nov 03 16:53:49.901129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.901105 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:49.929176 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:49.929146 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce4183a_b3a2_4f7f_a32d_4a59a37b6078.slice/crio-bfc09420b6dfdbfd7c29a832d0e6cb497508b78d76fab1ca700543c786237fd0 WatchSource:0}: Error finding container bfc09420b6dfdbfd7c29a832d0e6cb497508b78d76fab1ca700543c786237fd0: Status 404 returned error can't find the container with id bfc09420b6dfdbfd7c29a832d0e6cb497508b78d76fab1ca700543c786237fd0 Nov 03 16:53:49.930372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.930345 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 16:53:49.931772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.931739 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:49.931862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.931821 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:49.931862 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:49.931848 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:50.001393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.001040 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\") pod \"acaf0059-e09c-4b10-9a00-4971b7480aa0\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " Nov 03 16:53:50.001393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.001117 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\" (UniqueName: \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\") pod \"acaf0059-e09c-4b10-9a00-4971b7480aa0\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " Nov 03 16:53:50.001393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.001147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\") pod \"acaf0059-e09c-4b10-9a00-4971b7480aa0\" (UID: \"acaf0059-e09c-4b10-9a00-4971b7480aa0\") " Nov 03 16:53:50.002834 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.002169 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca" pod "acaf0059-e09c-4b10-9a00-4971b7480aa0" (UID: "acaf0059-e09c-4b10-9a00-4971b7480aa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:50.004621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.004593 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l" pod "acaf0059-e09c-4b10-9a00-4971b7480aa0" (UID: "acaf0059-e09c-4b10-9a00-4971b7480aa0"). InnerVolumeSpecName "kube-api-access-8m56l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:50.006394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.006370 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca" pod "acaf0059-e09c-4b10-9a00-4971b7480aa0" (UID: "acaf0059-e09c-4b10-9a00-4971b7480aa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:50.105494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.103624 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:50.105494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.103660 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m56l\" (UniqueName: \"kubernetes.io/projected/acaf0059-e09c-4b10-9a00-4971b7480aa0-kube-api-access-8m56l\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:50.105494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.103674 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acaf0059-e09c-4b10-9a00-4971b7480aa0-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:50.134002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.133961 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:53:50.134973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.134950 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 16:53:50.135059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.135000 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 16:53:50.687109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.686832 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" Nov 03 16:53:50.687109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.686852 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5" event={"ID":"acaf0059-e09c-4b10-9a00-4971b7480aa0","Type":"ContainerDied","Data":"1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1"} Nov 03 16:53:50.687109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.686925 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1709cd93ac0ebbb9f9806fef48509e86af20e77da451c35a6014371632955fb1" Nov 03 16:53:50.690340 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.690310 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" event={"ID":"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078","Type":"ContainerStarted","Data":"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca"} Nov 03 16:53:50.690517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.690353 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" event={"ID":"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078","Type":"ContainerStarted","Data":"bfc09420b6dfdbfd7c29a832d0e6cb497508b78d76fab1ca700543c786237fd0"} Nov 03 16:53:50.740415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:50.740365 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podStartSLOduration=1.740351848 podStartE2EDuration="1.740351848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:50.738213424 +0000 UTC m=+2326.071426984" watchObservedRunningTime="2026-11-03 16:53:50.740351848 +0000 UTC m=+2326.073565417" Nov 03 16:53:51.133499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:51.133463 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 16:53:51.133663 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:51.133520 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 16:53:51.596989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:51.596946 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:51.598688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:51.598664 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:51.598778 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:51.598722 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:52.134276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:52.134234 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 16:53:52.134492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:52.134310 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 16:53:52.299212 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:53:52.299168 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:53:52.597887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:52.597829 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:52.598426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:52.597926 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:53.133852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:53.133812 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 16:53:53.134062 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:53.133866 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 16:53:53.597383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:53.597349 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:53.597611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:53.597557 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:54.009976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.009929 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 16:53:54.010651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.010633 6001 cpu_manager.go:414] "containerMap: removing container" podUID="acaf0059-e09c-4b10-9a00-4971b7480aa0" containerName="revoke-keys" Nov 03 16:53:54.010651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.010651 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acaf0059-e09c-4b10-9a00-4971b7480aa0" containerName="revoke-keys" Nov 03 16:53:54.010770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.010761 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="acaf0059-e09c-4b10-9a00-4971b7480aa0" containerName="revoke-keys" Nov 03 16:53:54.013963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.013939 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.052468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.047829 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 16:53:54.106416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106265 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.106416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106386 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106425 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106474 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106601 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106638 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106687 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106721 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.107327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.106779 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2bt9\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.134470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.133734 6001 patch_prober.go:29] interesting pod/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Nov 03 16:53:54.134470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.133784 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Nov 03 16:53:54.208436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208373 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2bt9\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208628 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208696 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208672 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208704 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208876 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208839 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208879 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.208998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208958 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.209117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.208997 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.209531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.209501 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.210128 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.210097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.211150 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.211118 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.211280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.211236 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.211556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.211530 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.213157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.213129 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.213248 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.213213 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.233617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.233584 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2bt9\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.234532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.234510 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.332627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.331703 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:54.601338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.601236 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:54.601338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.601300 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:54.798561 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:54.798507 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a9efe6_aeb6_4161_8ec3_180c5b963891.slice/crio-8b7e16779d7f36889b421b3e971563329fecc7764486f075b91c8fc74e30cd1b WatchSource:0}: Error finding container 8b7e16779d7f36889b421b3e971563329fecc7764486f075b91c8fc74e30cd1b: Status 404 returned error can't find the container with id 8b7e16779d7f36889b421b3e971563329fecc7764486f075b91c8fc74e30cd1b Nov 03 16:53:54.800364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.800341 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 16:53:54.801000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.800970 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:54.801061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.801048 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:54.801107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.801081 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:53:54.907391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.907366 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:53:54.985375 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.985321 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn"] Nov 03 16:53:54.989322 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:54.989301 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.009003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.008972 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn"] Nov 03 16:53:55.009209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.009114 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:53:55.009209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.009159 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:53:55.009209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.009189 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:53:55.009338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.009207 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:53:55.127623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.127518 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.127623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.127577 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxxvd\" (UniqueName: \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.128056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.127830 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.138833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.138802 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:53:55.228934 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.228884 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.229133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.228941 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxxvd\" (UniqueName: \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.229133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.229130 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.229801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.229774 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.231539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.231504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.267470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.255217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxxvd\" (UniqueName: \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\") pod \"maas-api-revoke-sub-20fac21b-5zbtn\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.306931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.306892 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:55.599048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.599002 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:55.599256 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.599084 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:55.640656 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:55.640626 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edc192b_af7f_495e_8092_ca9facc4cbfc.slice/crio-bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e WatchSource:0}: Error finding container bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e: Status 404 returned error can't find the container with id bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e Nov 03 16:53:55.641199 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.641170 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn"] Nov 03 16:53:55.804413 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.804327 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" event={"ID":"1edc192b-af7f-495e-8092-ca9facc4cbfc","Type":"ContainerStarted","Data":"bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e"} Nov 03 16:53:55.807952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.807725 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" event={"ID":"07a9efe6-aeb6-4161-8ec3-180c5b963891","Type":"ContainerStarted","Data":"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4"} Nov 03 16:53:55.807952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.807809 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" event={"ID":"07a9efe6-aeb6-4161-8ec3-180c5b963891","Type":"ContainerStarted","Data":"8b7e16779d7f36889b421b3e971563329fecc7764486f075b91c8fc74e30cd1b"} Nov 03 16:53:55.809904 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.809840 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:53:55.876877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:55.867062 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podStartSLOduration=2.8670399509999998 podStartE2EDuration="2.867039951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:55.864769352 +0000 UTC m=+2331.197982922" watchObservedRunningTime="2026-11-03 16:53:55.867039951 +0000 UTC m=+2331.200253528" Nov 03 16:53:56.333278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.333185 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:53:56.334326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.334301 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.334423 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.334354 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:53:56.599587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.598973 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.599587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.599045 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:56.817081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.817044 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" event={"ID":"1edc192b-af7f-495e-8092-ca9facc4cbfc","Type":"ContainerStarted","Data":"0070c043facd9a5503e5cb47c2e810048f44b3cdbcffe56e6e0520a57a709aef"} Nov 03 16:53:56.880017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.879875 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" podStartSLOduration=2.879858685 podStartE2EDuration="2.879858685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:56.86372793 +0000 UTC m=+2332.196941503" watchObservedRunningTime="2026-11-03 16:53:56.879858685 +0000 UTC m=+2332.213072250" Nov 03 16:53:56.903313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.903280 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm"] Nov 03 16:53:56.907452 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.907419 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:56.928097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:56.928062 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm"] Nov 03 16:53:57.001381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.001346 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klxzn\" (UniqueName: \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.002112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.002082 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.002267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.002164 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.104046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.103999 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.104264 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.104112 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.104356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.104336 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klxzn\" (UniqueName: \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.105122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.105090 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.106590 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.106559 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.130782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.130702 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klxzn\" (UniqueName: \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\") pod \"maas-api-revoke-sub-e7d2f268-qzhbm\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.231171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.231133 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:53:57.340349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.340310 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.340782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.340377 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:53:57.575489 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.574107 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm"] Nov 03 16:53:57.580209 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:53:57.580182 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf47deb0b_4a5b_4feb_8a32_6c9325fe43d4.slice/crio-705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e WatchSource:0}: Error finding container 705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e: Status 404 returned error can't find the container with id 705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e Nov 03 16:53:57.597912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.597883 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.598029 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.597981 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Nov 03 16:53:57.610458 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.610408 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:53:57.610739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.610700 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" containerID="cri-o://c1cdfe84c524a535e17dc4fcc818b471a87a6b67f9d4a8dfb980f2403b6b5eb1" gracePeriod=30 Nov 03 16:53:57.620576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.620548 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:53:57.620785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.620762 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" containerID="cri-o://8b817491a87b3f3ea06f721c7753fa0e5060d4f4c37ed0dc8df8df3ecf34c099" gracePeriod=30 Nov 03 16:53:57.914497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.897126 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" event={"ID":"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4","Type":"ContainerStarted","Data":"705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e"} Nov 03 16:53:57.914497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.900889 6001 generic.go:356] "Generic (PLEG): container finished" podID="1edc192b-af7f-495e-8092-ca9facc4cbfc" containerID="0070c043facd9a5503e5cb47c2e810048f44b3cdbcffe56e6e0520a57a709aef" exitCode=0 Nov 03 16:53:57.914497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:57.901842 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" event={"ID":"1edc192b-af7f-495e-8092-ca9facc4cbfc","Type":"ContainerDied","Data":"0070c043facd9a5503e5cb47c2e810048f44b3cdbcffe56e6e0520a57a709aef"} Nov 03 16:53:58.335241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:58.335203 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:53:58.335784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:58.335750 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:53:58.976850 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:58.976814 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" event={"ID":"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4","Type":"ContainerStarted","Data":"8d4c52a5ee8c2328b206a341ab447eea5b0179359cf103eacb2c914197331273"} Nov 03 16:53:59.029071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.029010 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" podStartSLOduration=3.028988251 podStartE2EDuration="3.028988251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:59.025205625 +0000 UTC m=+2334.358419195" watchObservedRunningTime="2026-11-03 16:53:59.028988251 +0000 UTC m=+2334.362201881" Nov 03 16:53:59.089510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.087283 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:59.089510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.089185 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:59.090767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.090746 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 16:53:59.340583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.333326 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:53:59.340583 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.333411 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:53:59.535961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.533520 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:53:59.676554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.675042 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\") pod \"1edc192b-af7f-495e-8092-ca9facc4cbfc\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " Nov 03 16:53:59.676554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.675097 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\" (UniqueName: \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\") pod \"1edc192b-af7f-495e-8092-ca9facc4cbfc\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " Nov 03 16:53:59.676554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.675164 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\") pod \"1edc192b-af7f-495e-8092-ca9facc4cbfc\" (UID: \"1edc192b-af7f-495e-8092-ca9facc4cbfc\") " Nov 03 16:53:59.714041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.713977 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca" pod "1edc192b-af7f-495e-8092-ca9facc4cbfc" (UID: "1edc192b-af7f-495e-8092-ca9facc4cbfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:59.718666 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.718605 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd" pod "1edc192b-af7f-495e-8092-ca9facc4cbfc" (UID: "1edc192b-af7f-495e-8092-ca9facc4cbfc"). InnerVolumeSpecName "kube-api-access-cxxvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:59.719054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.719011 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca" pod "1edc192b-af7f-495e-8092-ca9facc4cbfc" (UID: "1edc192b-af7f-495e-8092-ca9facc4cbfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:53:59.805531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.804626 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:59.805531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.804666 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edc192b-af7f-495e-8092-ca9facc4cbfc-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:59.805531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.804682 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxxvd\" (UniqueName: \"kubernetes.io/projected/1edc192b-af7f-495e-8092-ca9facc4cbfc-kube-api-access-cxxvd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:53:59.990196 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.990156 6001 generic.go:356] "Generic (PLEG): container finished" podID="f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" containerID="8d4c52a5ee8c2328b206a341ab447eea5b0179359cf103eacb2c914197331273" exitCode=0 Nov 03 16:53:59.990903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.990873 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" event={"ID":"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4","Type":"ContainerDied","Data":"8d4c52a5ee8c2328b206a341ab447eea5b0179359cf103eacb2c914197331273"} Nov 03 16:53:59.994390 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:53:59.994363 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" Nov 03 16:54:00.003160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.003123 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn" event={"ID":"1edc192b-af7f-495e-8092-ca9facc4cbfc","Type":"ContainerDied","Data":"bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e"} Nov 03 16:54:00.003408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.003390 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf77bb21dc6221146059ae8507454ad049219cbb16fa45a63225ea085652ae8e" Nov 03 16:54:00.122306 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.120177 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s"] Nov 03 16:54:00.123836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.123806 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1edc192b-af7f-495e-8092-ca9facc4cbfc" containerName="revoke-keys" Nov 03 16:54:00.124499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.124421 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edc192b-af7f-495e-8092-ca9facc4cbfc" containerName="revoke-keys" Nov 03 16:54:00.127940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.127917 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edc192b-af7f-495e-8092-ca9facc4cbfc" containerName="revoke-keys" Nov 03 16:54:00.140828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.140789 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.338758 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.338633 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:54:00.458217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.455268 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:54:00.471377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.443943 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.475591 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.475547 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.475956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.475930 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2vt\" (UniqueName: \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.502059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.502027 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s"] Nov 03 16:54:00.585479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.585416 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.585847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.585819 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.586030 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.586012 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2vt\" (UniqueName: \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.603574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.588193 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.605073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.604983 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.645462 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.625911 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2vt\" (UniqueName: \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\") pod \"maas-api-revoke-sub-655bbfa7-kxs9s\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:00.771467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:00.770598 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:01.207076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.206554 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.378469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.377506 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:54:01.378469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.377576 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:54:01.424704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.415844 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.812574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.774240 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:54:01.812574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.774577 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:54:01.854517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.816778 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.854517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.816898 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.997581 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:01.970369 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.340568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.339705 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:54:02.362854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.349111 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" event={"ID":"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4","Type":"ContainerDied","Data":"705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e"} Nov 03 16:54:02.362854 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.349179 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="705a85172ea809c9e5d53fb6bcdfbb90b742fea720ac57feb87354df874a2a5e" Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.395597 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.395688 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.408667 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\" (UniqueName: \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\") pod \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.408768 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\") pod \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.408933 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\") pod \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\" (UID: \"f47deb0b-4a5b-4feb-8a32-6c9325fe43d4\") " Nov 03 16:54:02.434477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.417672 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca" pod "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" (UID: "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:02.506471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.456807 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 16:54:02.506471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.457946 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:54:02.535233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.507180 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.586833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.558233 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:02.880468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.855576 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.880468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.866226 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 16:54:02.880468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.866605 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.989516 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:02.971129 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:03.049492 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:02.997980 6001 container.go:588] Failed to update stats for container "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da63c2_ca5a_4a9f_b177_54a511c9a790.slice/crio-14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da63c2_ca5a_4a9f_b177_54a511c9a790.slice/crio-14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da63c2_ca5a_4a9f_b177_54a511c9a790.slice/crio-14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956/memory.stat: no such device], continuing to push stats Nov 03 16:54:03.191620 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:03.176882 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40d3cc9_878a_4784_822b_a2eda88ff49d.slice/crio-af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca WatchSource:0}: Error finding container af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca: Status 404 returned error can't find the container with id af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca Nov 03 16:54:03.216717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.216678 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca" pod "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" (UID: "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:03.221334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.221278 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn" pod "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" (UID: "f47deb0b-4a5b-4feb-8a32-6c9325fe43d4"). InnerVolumeSpecName "kube-api-access-klxzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:03.238585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.238539 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s"] Nov 03 16:54:03.238914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.238867 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:54:03.239098 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.239082 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:54:03.239991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.239288 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:54:03.240175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240157 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:54:03.240345 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240331 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 16:54:03.240498 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240485 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:54:03.240615 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240604 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:54:03.240724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240713 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:54:03.240870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.240858 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:54:03.290765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.283412 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klxzn\" (UniqueName: \"kubernetes.io/projected/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-kube-api-access-klxzn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.290765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.283502 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.330179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.330137 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:54:03.331525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.331494 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" containerName="revoke-keys" Nov 03 16:54:03.331715 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.331699 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" containerName="revoke-keys" Nov 03 16:54:03.332046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.332030 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" containerName="revoke-keys" Nov 03 16:54:03.337435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.334438 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Nov 03 16:54:03.337435 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.334518 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Nov 03 16:54:03.339255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.339213 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.352202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.352078 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:54:03.372959 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.372919 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" event={"ID":"f40d3cc9-878a-4784-822b-a2eda88ff49d","Type":"ContainerStarted","Data":"af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca"} Nov 03 16:54:03.375249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.374849 6001 generic.go:356] "Generic (PLEG): container finished" podID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerID="8b817491a87b3f3ea06f721c7753fa0e5060d4f4c37ed0dc8df8df3ecf34c099" exitCode=0 Nov 03 16:54:03.375249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.374962 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" event={"ID":"89da63c2-ca5a-4a9f-b177-54a511c9a790","Type":"ContainerDied","Data":"8b817491a87b3f3ea06f721c7753fa0e5060d4f4c37ed0dc8df8df3ecf34c099"} Nov 03 16:54:03.378429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.377372 6001 generic.go:356] "Generic (PLEG): container finished" podID="81ee91dd-8787-42b8-b981-94782d875b3d" containerID="c1cdfe84c524a535e17dc4fcc818b471a87a6b67f9d4a8dfb980f2403b6b5eb1" exitCode=0 Nov 03 16:54:03.378429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.377633 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm" Nov 03 16:54:03.378429 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.377663 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" event={"ID":"81ee91dd-8787-42b8-b981-94782d875b3d","Type":"ContainerDied","Data":"c1cdfe84c524a535e17dc4fcc818b471a87a6b67f9d4a8dfb980f2403b6b5eb1"} Nov 03 16:54:03.488415 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.488380 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.488652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.488630 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.488765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.488665 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr955\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.488765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.488728 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.488936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.488788 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.489190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.489159 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.489376 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.489352 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.489532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.489502 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.489627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.489550 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.551797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.551767 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:54:03.577613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.577526 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:54:03.594521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.594482 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.594719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.594575 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.594719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.594607 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr955\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.595084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.594978 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.595202 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.595151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.595273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.595223 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.595689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.595658 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596182 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596293 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596483 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596615 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596647 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.596706 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.596654 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.597622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.597273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.599616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.599592 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.599959 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.599938 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.619343 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.619311 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr955\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.622840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.622811 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.675630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.675595 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697417 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697561 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697644 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697681 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697714 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697750 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697782 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697809 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697834 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697858 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697884 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697916 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697940 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697957 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.697978 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.698065 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.698090 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\") pod \"89da63c2-ca5a-4a9f-b177-54a511c9a790\" (UID: \"89da63c2-ca5a-4a9f-b177-54a511c9a790\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.698154 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\") pod \"81ee91dd-8787-42b8-b981-94782d875b3d\" (UID: \"81ee91dd-8787-42b8-b981-94782d875b3d\") " Nov 03 16:54:03.699107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.698904 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:03.700296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.699182 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.700296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.700099 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.700408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.700325 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.701744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.700887 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.701744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.701232 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.701744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.701432 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.701744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.701583 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.701744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.701629 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.702185 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.702089 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:03.702243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.702181 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:54:03.713657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.711812 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:54:03.713657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.711871 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "kube-api-access-rmclj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:03.713657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.712008 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.714041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.713987 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:03.714041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.713997 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "kube-api-access-lsmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:03.715106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.715078 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token" pod "81ee91dd-8787-42b8-b981-94782d875b3d" (UID: "81ee91dd-8787-42b8-b981-94782d875b3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:03.716147 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.716111 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token" pod "89da63c2-ca5a-4a9f-b177-54a511c9a790" (UID: "89da63c2-ca5a-4a9f-b177-54a511c9a790"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799814 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81ee91dd-8787-42b8-b981-94782d875b3d-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799863 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799880 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmclj\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-kube-api-access-rmclj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799896 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89da63c2-ca5a-4a9f-b177-54a511c9a790-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799910 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799942 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799955 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799969 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.799983 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ee91dd-8787-42b8-b981-94782d875b3d-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800016 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800029 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89da63c2-ca5a-4a9f-b177-54a511c9a790-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800041 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800059 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800092 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800105 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsmn7\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-kube-api-access-lsmn7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800119 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89da63c2-ca5a-4a9f-b177-54a511c9a790-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800135 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81ee91dd-8787-42b8-b981-94782d875b3d-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:03.800162 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:03.800147 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81ee91dd-8787-42b8-b981-94782d875b3d-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:04.041125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.041087 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:54:04.045681 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:04.045643 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a52f3a_2bd0_4bdf_9181_9719d241a0f7.slice/crio-1b7d57e8c8f4e2221798d0e6577b3daa92e1b9d608e494b4e2e1575e2b849822 WatchSource:0}: Error finding container 1b7d57e8c8f4e2221798d0e6577b3daa92e1b9d608e494b4e2e1575e2b849822: Status 404 returned error can't find the container with id 1b7d57e8c8f4e2221798d0e6577b3daa92e1b9d608e494b4e2e1575e2b849822 Nov 03 16:54:04.048741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.048700 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:04.048886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.048796 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:04.048886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.048842 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:04.168335 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.168280 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 16:54:04.169354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169327 6001 cpu_manager.go:414] "containerMap: removing container" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" Nov 03 16:54:04.169513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169357 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" Nov 03 16:54:04.169513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169377 6001 cpu_manager.go:414] "containerMap: removing container" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" Nov 03 16:54:04.169513 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169386 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" Nov 03 16:54:04.169717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169606 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" containerName="istio-proxy" Nov 03 16:54:04.169717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.169623 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" containerName="istio-proxy" Nov 03 16:54:04.174189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.174159 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.191885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.191836 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 16:54:04.212603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.212537 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.212878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.212851 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.213061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.213044 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.213193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.213179 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjbn\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.213380 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.213359 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.213659 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.213635 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.213937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.213895 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.214096 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.214078 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.214262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.214239 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.214418 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.214350 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.214864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.214144 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 16:54:04.260512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.253315 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 16:54:04.319896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.319797 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.319896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.319852 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.319896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.319887 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.319966 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320007 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320034 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320076 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320112 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320138 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.320183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320168 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.320584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320254 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320278 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.320584 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320305 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320696 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320740 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320708 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.320968 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.320941 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.321176 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.321152 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.321273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.321254 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.321608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.321587 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.321733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.321615 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qp4k\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.321892 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.321874 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjbn\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.322072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.322054 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.322637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.322614 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.325140 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.325113 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.326640 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.326622 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.346739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.346701 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjbn\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.352580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.352541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.396387 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.396303 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" event={"ID":"85a52f3a-2bd0-4bdf-9181-9719d241a0f7","Type":"ContainerStarted","Data":"24ca42c1e3a2b597730f56af84e94bdc834e3e67e640b7fba3769517ea83d8de"} Nov 03 16:54:04.397178 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.397146 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" event={"ID":"85a52f3a-2bd0-4bdf-9181-9719d241a0f7","Type":"ContainerStarted","Data":"1b7d57e8c8f4e2221798d0e6577b3daa92e1b9d608e494b4e2e1575e2b849822"} Nov 03 16:54:04.402033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.401992 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" event={"ID":"f40d3cc9-878a-4784-822b-a2eda88ff49d","Type":"ContainerStarted","Data":"10caff164cbd8f6b24f778c3fc022f918a158bf7ed3e07a9c1e1df2a85d42f57"} Nov 03 16:54:04.404265 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.404237 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" Nov 03 16:54:04.404426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.404269 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5" event={"ID":"89da63c2-ca5a-4a9f-b177-54a511c9a790","Type":"ContainerDied","Data":"14d7d25868dea7451917535d1ae0489db0bc8c98b875053ebf1f4474d4aec956"} Nov 03 16:54:04.404426 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.404358 6001 scope.go:122] "RemoveContainer" containerID="8b817491a87b3f3ea06f721c7753fa0e5060d4f4c37ed0dc8df8df3ecf34c099" Nov 03 16:54:04.406334 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.406305 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" event={"ID":"81ee91dd-8787-42b8-b981-94782d875b3d","Type":"ContainerDied","Data":"44ff582510692af0fb24652345d07cafad79437adf202d04d2ca187c3f3810d8"} Nov 03 16:54:04.406518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.406351 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8" Nov 03 16:54:04.419254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.419235 6001 scope.go:122] "RemoveContainer" containerID="c1cdfe84c524a535e17dc4fcc818b471a87a6b67f9d4a8dfb980f2403b6b5eb1" Nov 03 16:54:04.423729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.423691 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.423878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.423839 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.423955 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.423930 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.424021 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.423990 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qp4k\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424087 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424121 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424185 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424335 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424388 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424438 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.424486 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.425038 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.425225 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.425724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.425277 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.426899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.426876 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.427109 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.427057 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.449729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.449663 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podStartSLOduration=1.449640759 podStartE2EDuration="1.449640759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:04.447752951 +0000 UTC m=+2339.780966521" watchObservedRunningTime="2026-11-03 16:54:04.449640759 +0000 UTC m=+2339.782854328" Nov 03 16:54:04.451363 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.450902 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qp4k\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.456157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.456117 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.493171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.493081 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" podStartSLOduration=4.493064989 podStartE2EDuration="4.493064989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:04.49139228 +0000 UTC m=+2339.824605870" watchObservedRunningTime="2026-11-03 16:54:04.493064989 +0000 UTC m=+2339.826278610" Nov 03 16:54:04.520777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.520741 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:04.556121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.556090 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:04.589258 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.589213 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:54:04.607846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.607803 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jn8"] Nov 03 16:54:04.628678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.628640 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:54:04.638691 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.638552 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mk2z5"] Nov 03 16:54:04.676477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.676424 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:04.808515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.804934 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ee91dd-8787-42b8-b981-94782d875b3d" path="/var/lib/kubelet/pods/81ee91dd-8787-42b8-b981-94782d875b3d/volumes" Nov 03 16:54:04.808515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.808293 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89da63c2-ca5a-4a9f-b177-54a511c9a790" path="/var/lib/kubelet/pods/89da63c2-ca5a-4a9f-b177-54a511c9a790/volumes" Nov 03 16:54:04.844604 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.844523 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 16:54:04.851922 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.851888 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:04.882739 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.882700 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3dd067-dockercfg-ztjgk\"" Nov 03 16:54:04.882945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.882808 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3dd067\"" Nov 03 16:54:04.888249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.888200 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 16:54:04.906409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.906371 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 16:54:04.910993 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.910934 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:04.932572 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:04.932529 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 16:54:05.028332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.023001 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 16:54:05.054056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.040010 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.054056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.040095 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrjxl\" (UniqueName: \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.054056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.040154 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.054056 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.040250 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nhgt\" (UniqueName: \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.143370 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nhgt\" (UniqueName: \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.143630 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.143705 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrjxl\" (UniqueName: \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.143771 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.144832 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.153195 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.146000 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.187474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.158611 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.187474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.158715 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.187474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.158758 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.207475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.199194 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 16:54:05.209229 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.209191 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.209671 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.209638 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.209893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.209857 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:05.229351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.229320 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrjxl\" (UniqueName: \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.277496 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.265278 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nhgt\" (UniqueName: \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\") pod \"payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.374468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.373628 6001 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:54:05.374468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.373703 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:05.422042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.422004 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" event={"ID":"da584ac1-244e-4ce8-bf4e-819777f46f6d","Type":"ContainerStarted","Data":"7ead556c62d0a241376e1feefe0ccd2ec8de8e5ea9ef743fa7ee1b88d357483c"} Nov 03 16:54:05.424333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.424282 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" event={"ID":"95e48b6d-a1ab-43e0-888a-8549109703fc","Type":"ContainerStarted","Data":"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec"} Nov 03 16:54:05.424333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.424332 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" event={"ID":"95e48b6d-a1ab-43e0-888a-8549109703fc","Type":"ContainerStarted","Data":"06aed220ab853848ea5f9366bc1fa76bade6c64607f2157e16b1e306b6b5d9ab"} Nov 03 16:54:05.426367 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.426333 6001 generic.go:356] "Generic (PLEG): container finished" podID="f40d3cc9-878a-4784-822b-a2eda88ff49d" containerID="10caff164cbd8f6b24f778c3fc022f918a158bf7ed3e07a9c1e1df2a85d42f57" exitCode=0 Nov 03 16:54:05.426554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.426384 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" event={"ID":"f40d3cc9-878a-4784-822b-a2eda88ff49d","Type":"ContainerDied","Data":"10caff164cbd8f6b24f778c3fc022f918a158bf7ed3e07a9c1e1df2a85d42f57"} Nov 03 16:54:05.501471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.501416 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:05.514764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.514697 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podStartSLOduration=1.514675877 podStartE2EDuration="1.514675877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:05.496002308 +0000 UTC m=+2340.829215878" watchObservedRunningTime="2026-11-03 16:54:05.514675877 +0000 UTC m=+2340.847889448" Nov 03 16:54:05.515127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.515095 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:05.527809 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.527723 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:05.722243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.722132 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:05.722243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:05.722189 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:06.093479 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.072752 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:54:06.095205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.095122 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:54:06.098061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.098002 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 16:54:06.243567 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.243375 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 16:54:06.530618 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.515590 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 16:54:06.540894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.540862 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.541152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.541127 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:06.635283 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.635243 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" event={"ID":"6274d78b-7ebd-493e-9712-904c7320f5de","Type":"ContainerStarted","Data":"826579c738753597d25cd4a1794c83eaa584c015988fd3ee208608df815b39fe"} Nov 03 16:54:06.638039 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.637995 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" event={"ID":"da584ac1-244e-4ce8-bf4e-819777f46f6d","Type":"ContainerStarted","Data":"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1"} Nov 03 16:54:06.640286 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.640253 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" event={"ID":"46d2b661-ff65-486a-aa75-5e27fc535040","Type":"ContainerStarted","Data":"d5cdc147e310eb7e6a6dbb15680b34ab17fffd96bf32547eac2f4d32cca348cd"} Nov 03 16:54:06.727468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.727031 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.727468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.727094 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:06.935701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:06.935615 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podStartSLOduration=2.935560109 podStartE2EDuration="2.935560109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:06.760812103 +0000 UTC m=+2342.094025675" watchObservedRunningTime="2026-11-03 16:54:06.935560109 +0000 UTC m=+2342.268773681" Nov 03 16:54:07.532002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.531963 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.534154 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.534112 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:07.618963 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.618929 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:07.724507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.629136 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.724507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.691905 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:07.724507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.687278 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.724507 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.691988 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:07.943431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.943341 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" event={"ID":"46d2b661-ff65-486a-aa75-5e27fc535040","Type":"ContainerStarted","Data":"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5"} Nov 03 16:54:07.947658 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:07.947629 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:08.035484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.031544 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" podStartSLOduration=4.03152587 podStartE2EDuration="4.03152587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:08.013693994 +0000 UTC m=+2343.346907565" watchObservedRunningTime="2026-11-03 16:54:08.03152587 +0000 UTC m=+2343.364739439" Nov 03 16:54:08.041012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.040976 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" event={"ID":"6274d78b-7ebd-493e-9712-904c7320f5de","Type":"ContainerStarted","Data":"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0"} Nov 03 16:54:08.041781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.041750 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:08.042742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.042722 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:08.105545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.102379 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\") pod \"f40d3cc9-878a-4784-822b-a2eda88ff49d\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " Nov 03 16:54:08.105545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.102468 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\") pod \"f40d3cc9-878a-4784-822b-a2eda88ff49d\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " Nov 03 16:54:08.105545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.102782 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\" (UniqueName: \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\") pod \"f40d3cc9-878a-4784-822b-a2eda88ff49d\" (UID: \"f40d3cc9-878a-4784-822b-a2eda88ff49d\") " Nov 03 16:54:08.107901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.107850 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca" pod "f40d3cc9-878a-4784-822b-a2eda88ff49d" (UID: "f40d3cc9-878a-4784-822b-a2eda88ff49d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:08.130664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.122951 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca" pod "f40d3cc9-878a-4784-822b-a2eda88ff49d" (UID: "f40d3cc9-878a-4784-822b-a2eda88ff49d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:08.130664 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.126083 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt" pod "f40d3cc9-878a-4784-822b-a2eda88ff49d" (UID: "f40d3cc9-878a-4784-822b-a2eda88ff49d"). InnerVolumeSpecName "kube-api-access-9r2vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:08.133532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.133500 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2vt\" (UniqueName: \"kubernetes.io/projected/f40d3cc9-878a-4784-822b-a2eda88ff49d-kube-api-access-9r2vt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:08.133532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.133534 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:08.133751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.133548 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40d3cc9-878a-4784-822b-a2eda88ff49d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:08.136810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.136755 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" podStartSLOduration=4.136739064 podStartE2EDuration="4.136739064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:08.118429633 +0000 UTC m=+2343.451643203" watchObservedRunningTime="2026-11-03 16:54:08.136739064 +0000 UTC m=+2343.469952633" Nov 03 16:54:08.528471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.522923 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.528471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.522992 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:08.610473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.610386 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.611023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.610567 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:08.697465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.693993 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.697465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:08.694057 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:09.141466 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.140430 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" Nov 03 16:54:09.158361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.148604 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s" event={"ID":"f40d3cc9-878a-4784-822b-a2eda88ff49d","Type":"ContainerDied","Data":"af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca"} Nov 03 16:54:09.158361 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.148681 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4037145b01575ea08dd3345281360e5b2f1a133111b9af20b88a884bf4fcca" Nov 03 16:54:09.540987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.538589 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.540987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.538676 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:09.663472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.662821 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.663472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.663004 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:09.705471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.699240 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.705471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:09.699325 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:10.523468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.512506 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d"] Nov 03 16:54:10.523468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.513890 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f40d3cc9-878a-4784-822b-a2eda88ff49d" containerName="revoke-keys" Nov 03 16:54:10.523468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.513929 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40d3cc9-878a-4784-822b-a2eda88ff49d" containerName="revoke-keys" Nov 03 16:54:10.523468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.514154 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40d3cc9-878a-4784-822b-a2eda88ff49d" containerName="revoke-keys" Nov 03 16:54:10.532832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.530080 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.532832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.530143 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:10.538792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.538338 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d"] Nov 03 16:54:10.538792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.538478 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.588465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.587681 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.588465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.587750 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:10.607588 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.605857 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:54:10.619838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.593234 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:54:10.619998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.619954 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:54:10.633627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.633597 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:54:10.688487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.688439 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.689112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.689056 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:10.715355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.715313 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.717468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.715759 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fflhq\" (UniqueName: \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.717708 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.717684 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.828536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.827605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.828536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.827780 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fflhq\" (UniqueName: \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.828536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.827826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.849470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.836002 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.907472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.882512 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.929470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.922268 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fflhq\" (UniqueName: \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\") pod \"maas-api-revoke-sub-54eaf8bd-svg8d\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:10.929931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:10.929896 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:11.333643 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.333594 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z"] Nov 03 16:54:11.338377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.337986 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z"] Nov 03 16:54:11.338377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.338126 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.420249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.420155 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.420751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.420722 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.428599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.428561 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9956\" (UniqueName: \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.527492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.527034 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.527492 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.527100 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:11.542278 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.531407 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9956\" (UniqueName: \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.573683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.573632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.591482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.583854 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.608164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.539270 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d"] Nov 03 16:54:11.608164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.587929 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.608164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.589565 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.608164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.607414 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:11.608164 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.606691 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.620899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.620850 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9956\" (UniqueName: \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\") pod \"maas-api-revoke-sub-8c1bc435-s5x5z\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:11.679527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.676499 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.679527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.676559 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Nov 03 16:54:11.835542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:11.835503 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:12.312976 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.312926 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" event={"ID":"daaa5361-c760-4b89-ab2c-7e43e01967ff","Type":"ContainerStarted","Data":"de4d04ea7156754542e2645dde770fe27443f44f1388352468fd87dc5293860b"} Nov 03 16:54:12.315081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.315046 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" event={"ID":"daaa5361-c760-4b89-ab2c-7e43e01967ff","Type":"ContainerStarted","Data":"632987f88a22a3ca1f0709b2637a643cacd1edf2261ae35862e84807562d3053"} Nov 03 16:54:12.454208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.454139 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" podStartSLOduration=2.454116322 podStartE2EDuration="2.454116322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:12.368583065 +0000 UTC m=+2347.701796636" watchObservedRunningTime="2026-11-03 16:54:12.454116322 +0000 UTC m=+2347.787329893" Nov 03 16:54:12.457898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.457815 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z"] Nov 03 16:54:12.458427 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:12.458394 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b7d753_c42b_4903_bbe9_daa084e1f230.slice/crio-60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8 WatchSource:0}: Error finding container 60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8: Status 404 returned error can't find the container with id 60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8 Nov 03 16:54:12.522689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.522644 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body= Nov 03 16:54:12.522919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.522718 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" Nov 03 16:54:12.581524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.572249 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Nov 03 16:54:12.581524 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.572336 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Nov 03 16:54:12.876349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.876307 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:12.876919 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.876814 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:12.877699 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:12.877670 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:54:13.327522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.326660 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" event={"ID":"78b7d753-c42b-4903-bbe9-daa084e1f230","Type":"ContainerStarted","Data":"c560dd8058f069c0fa380781f8b0525bee4ffddcf121a9ba5025e5ac76e045ea"} Nov 03 16:54:13.327522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.327064 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" event={"ID":"78b7d753-c42b-4903-bbe9-daa084e1f230","Type":"ContainerStarted","Data":"60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8"} Nov 03 16:54:13.329549 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.329513 6001 generic.go:356] "Generic (PLEG): container finished" podID="daaa5361-c760-4b89-ab2c-7e43e01967ff" containerID="de4d04ea7156754542e2645dde770fe27443f44f1388352468fd87dc5293860b" exitCode=0 Nov 03 16:54:13.329840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.329554 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" event={"ID":"daaa5361-c760-4b89-ab2c-7e43e01967ff","Type":"ContainerDied","Data":"de4d04ea7156754542e2645dde770fe27443f44f1388352468fd87dc5293860b"} Nov 03 16:54:13.371905 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.371832 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" podStartSLOduration=2.371810179 podStartE2EDuration="2.371810179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:13.370916635 +0000 UTC m=+2348.704130205" watchObservedRunningTime="2026-11-03 16:54:13.371810179 +0000 UTC m=+2348.705023748" Nov 03 16:54:13.832838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.832802 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:13.834635 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.834602 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:13.835505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.835430 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 16:54:13.841270 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.841241 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:13.841653 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.841624 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:13.842613 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:13.842594 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 16:54:14.340397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.340353 6001 generic.go:356] "Generic (PLEG): container finished" podID="78b7d753-c42b-4903-bbe9-daa084e1f230" containerID="c560dd8058f069c0fa380781f8b0525bee4ffddcf121a9ba5025e5ac76e045ea" exitCode=0 Nov 03 16:54:14.340930 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.340483 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" event={"ID":"78b7d753-c42b-4903-bbe9-daa084e1f230","Type":"ContainerDied","Data":"c560dd8058f069c0fa380781f8b0525bee4ffddcf121a9ba5025e5ac76e045ea"} Nov 03 16:54:14.535433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.535397 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:14.658526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.658428 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\") pod \"daaa5361-c760-4b89-ab2c-7e43e01967ff\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " Nov 03 16:54:14.658707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.658577 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\" (UniqueName: \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\") pod \"daaa5361-c760-4b89-ab2c-7e43e01967ff\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " Nov 03 16:54:14.658707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.658647 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\") pod \"daaa5361-c760-4b89-ab2c-7e43e01967ff\" (UID: \"daaa5361-c760-4b89-ab2c-7e43e01967ff\") " Nov 03 16:54:14.658901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.658804 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca" pod "daaa5361-c760-4b89-ab2c-7e43e01967ff" (UID: "daaa5361-c760-4b89-ab2c-7e43e01967ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:14.659327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.659306 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:14.660679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.660648 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca" pod "daaa5361-c760-4b89-ab2c-7e43e01967ff" (UID: "daaa5361-c760-4b89-ab2c-7e43e01967ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:14.660842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.660819 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq" pod "daaa5361-c760-4b89-ab2c-7e43e01967ff" (UID: "daaa5361-c760-4b89-ab2c-7e43e01967ff"). InnerVolumeSpecName "kube-api-access-fflhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:14.760650 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.760611 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fflhq\" (UniqueName: \"kubernetes.io/projected/daaa5361-c760-4b89-ab2c-7e43e01967ff-kube-api-access-fflhq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:14.760865 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:14.760662 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daaa5361-c760-4b89-ab2c-7e43e01967ff-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:15.353245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.353211 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" Nov 03 16:54:15.353245 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.353227 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d" event={"ID":"daaa5361-c760-4b89-ab2c-7e43e01967ff","Type":"ContainerDied","Data":"632987f88a22a3ca1f0709b2637a643cacd1edf2261ae35862e84807562d3053"} Nov 03 16:54:15.353824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.353315 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="632987f88a22a3ca1f0709b2637a643cacd1edf2261ae35862e84807562d3053" Nov 03 16:54:15.537310 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.537277 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:15.673939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.673832 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\") pod \"78b7d753-c42b-4903-bbe9-daa084e1f230\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " Nov 03 16:54:15.673939 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.673925 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\" (UniqueName: \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\") pod \"78b7d753-c42b-4903-bbe9-daa084e1f230\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " Nov 03 16:54:15.674151 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.673965 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\") pod \"78b7d753-c42b-4903-bbe9-daa084e1f230\" (UID: \"78b7d753-c42b-4903-bbe9-daa084e1f230\") " Nov 03 16:54:15.674235 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.674210 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca" pod "78b7d753-c42b-4903-bbe9-daa084e1f230" (UID: "78b7d753-c42b-4903-bbe9-daa084e1f230"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:15.674515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.674498 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:15.676142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.676109 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca" pod "78b7d753-c42b-4903-bbe9-daa084e1f230" (UID: "78b7d753-c42b-4903-bbe9-daa084e1f230"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:15.676279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.676252 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956" pod "78b7d753-c42b-4903-bbe9-daa084e1f230" (UID: "78b7d753-c42b-4903-bbe9-daa084e1f230"). InnerVolumeSpecName "kube-api-access-s9956". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:15.776304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.776256 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9956\" (UniqueName: \"kubernetes.io/projected/78b7d753-c42b-4903-bbe9-daa084e1f230-kube-api-access-s9956\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:15.776304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:15.776294 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b7d753-c42b-4903-bbe9-daa084e1f230-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:16.363473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:16.363382 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" Nov 03 16:54:16.363473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:16.363422 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z" event={"ID":"78b7d753-c42b-4903-bbe9-daa084e1f230","Type":"ContainerDied","Data":"60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8"} Nov 03 16:54:16.363872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:16.363495 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60cfb9c5fe0d5c85cdf66c9797f88701ea24a767b4c3d8917c3a49d5cbe908e8" Nov 03 16:54:17.188821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.188789 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 16:54:17.189477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189437 6001 cpu_manager.go:414] "containerMap: removing container" podUID="78b7d753-c42b-4903-bbe9-daa084e1f230" containerName="revoke-keys" Nov 03 16:54:17.189477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189467 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b7d753-c42b-4903-bbe9-daa084e1f230" containerName="revoke-keys" Nov 03 16:54:17.189550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189508 6001 cpu_manager.go:414] "containerMap: removing container" podUID="daaa5361-c760-4b89-ab2c-7e43e01967ff" containerName="revoke-keys" Nov 03 16:54:17.189550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189514 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaa5361-c760-4b89-ab2c-7e43e01967ff" containerName="revoke-keys" Nov 03 16:54:17.189628 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189618 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaa5361-c760-4b89-ab2c-7e43e01967ff" containerName="revoke-keys" Nov 03 16:54:17.189661 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.189635 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b7d753-c42b-4903-bbe9-daa084e1f230" containerName="revoke-keys" Nov 03 16:54:17.192190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.192174 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.214397 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.214358 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3b6f35\"" Nov 03 16:54:17.226745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.226717 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3b6f35-dockercfg-g6lrb\"" Nov 03 16:54:17.232797 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.232751 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 16:54:17.323471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.295961 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cfv\" (UniqueName: \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.323471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.296143 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.397465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.397410 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cfv\" (UniqueName: \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.400996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.400962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.402437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.402396 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.466505 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.466277 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cfv\" (UniqueName: \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\") pod \"payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.506856 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.506818 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:17.814897 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:17.814851 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdc4bde_4c9c_4ae4_8953_da05fb0a6eb2.slice/crio-b20c022683131a12cc373bff684460d65caf8afee5c6c6a141e8f8f6b197c375 WatchSource:0}: Error finding container b20c022683131a12cc373bff684460d65caf8afee5c6c6a141e8f8f6b197c375: Status 404 returned error can't find the container with id b20c022683131a12cc373bff684460d65caf8afee5c6c6a141e8f8f6b197c375 Nov 03 16:54:17.815586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:17.815556 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 16:54:18.450476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:18.449433 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" event={"ID":"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2","Type":"ContainerStarted","Data":"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974"} Nov 03 16:54:18.450476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:18.450147 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:18.450476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:18.450184 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" event={"ID":"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2","Type":"ContainerStarted","Data":"b20c022683131a12cc373bff684460d65caf8afee5c6c6a141e8f8f6b197c375"} Nov 03 16:54:18.495769 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:18.495692 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" podStartSLOduration=1.49567105 podStartE2EDuration="1.49567105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:18.493331544 +0000 UTC m=+2353.826545173" watchObservedRunningTime="2026-11-03 16:54:18.49567105 +0000 UTC m=+2353.828884620" Nov 03 16:54:19.153383 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:19.153347 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 16:54:19.153621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:19.153429 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 16:54:22.616473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.592630 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 16:54:22.771121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.767539 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.954586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.894587 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.954586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.933572 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.954586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.933937 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 16:54:22.954586 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:22.936238 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.077781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.054623 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.083036 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:54:23.083000 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:54:23.095938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.092646 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-5a5ab1-openshift-default-dockercfg-n5kkj\"" Nov 03 16:54:23.211568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.198399 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.211568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.209018 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrv4w\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.211568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.209135 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.211568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.209219 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.235152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.235110 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.236325 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.235509 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.236568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.236529 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.236806 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.236786 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.237081 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.237047 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.237302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.237283 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.350741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.350667 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393227 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393392 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrv4w\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393477 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393552 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393604 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393684 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393720 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.393791 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.401836 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.356728 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.402233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.391807 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.402893 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.414504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.403602 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.477544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.445075 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.512302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.485844 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.512302 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.490417 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrv4w\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.689497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.544937 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.689497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.646015 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\") pod \"e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.734576 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.733837 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:23.773023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.768569 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.773023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.771377 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:54:23.901464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.897976 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 16:54:23.905465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.903933 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.928764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.912250 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.935778 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.938826 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.939272 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.939334 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.956694 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:23.980219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:23.958583 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:54:24.030627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.030512 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 16:54:24.053807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.053758 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:24.217623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.217583 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 16:54:24.225249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.224368 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 16:54:24.299874 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.299791 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 16:54:24.692107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.692076 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 16:54:24.694497 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:24.694469 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b830bc_c904_47e8_a5a3_f33cfaafff32.slice/crio-5d86678de4de5940ea5f5c04b9092856cb2b9a322e30d9b4103f875b05c93220 WatchSource:0}: Error finding container 5d86678de4de5940ea5f5c04b9092856cb2b9a322e30d9b4103f875b05c93220: Status 404 returned error can't find the container with id 5d86678de4de5940ea5f5c04b9092856cb2b9a322e30d9b4103f875b05c93220 Nov 03 16:54:24.698214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.698171 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:24.698326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.698264 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:24.698326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:24.698308 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:25.338885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:25.338845 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerStarted","Data":"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a"} Nov 03 16:54:25.339077 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:25.338905 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerStarted","Data":"5d86678de4de5940ea5f5c04b9092856cb2b9a322e30d9b4103f875b05c93220"} Nov 03 16:54:25.388364 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:25.388302 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podStartSLOduration=3.388286939 podStartE2EDuration="3.388286939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:25.386576073 +0000 UTC m=+2360.719789639" watchObservedRunningTime="2026-11-03 16:54:25.388286939 +0000 UTC m=+2360.721500623" Nov 03 16:54:25.735515 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:25.735479 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:25.751925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:25.751880 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:26.348802 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:26.348771 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:26.349838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:26.349819 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:54:26.914552 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:26.914510 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:54:26.914990 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:26.914775 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" containerID="cri-o://0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3" gracePeriod=30 Nov 03 16:54:29.461595 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:29.461566 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 16:54:31.179553 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.179502 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 16:54:31.264510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.257951 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 16:54:31.264510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.258085 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.275562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.275534 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5a5ab1\"" Nov 03 16:54:31.276054 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.276035 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5a5ab1-dockercfg-q9t7m\"" Nov 03 16:54:31.407379 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.407341 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2f68\" (UniqueName: \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.407627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.407562 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.509139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.509098 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.509336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.509296 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2f68\" (UniqueName: \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.509921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.509898 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.534318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.534285 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2f68\" (UniqueName: \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\") pod \"payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.584685 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.584638 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:31.906882 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:31.906837 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8d8b98_9f22_4f9e_9bc1_871e9d02bfc4.slice/crio-0cbdb9d07637b34014fa587e6681edaab70e8176271aa94f770cbad3ded19010 WatchSource:0}: Error finding container 0cbdb9d07637b34014fa587e6681edaab70e8176271aa94f770cbad3ded19010: Status 404 returned error can't find the container with id 0cbdb9d07637b34014fa587e6681edaab70e8176271aa94f770cbad3ded19010 Nov 03 16:54:31.908243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:31.908219 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 16:54:32.208373 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.208342 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:54:32.319705 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.319665 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.319942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.319743 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.319942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.319791 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.320534 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.321154 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.321764 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.321829 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.321984 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.322064 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.322200 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.323467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.322219 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:32.325494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.324054 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\") pod \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\" (UID: \"eb7e9ee8-8c83-47b5-b501-d71c503fca13\") " Nov 03 16:54:32.325494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.324823 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:32.325494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.325160 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:32.326070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.326028 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "kube-api-access-jnks5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:32.326885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.326850 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:54:32.327026 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327008 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327034 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327048 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327062 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327077 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327091 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnks5\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-kube-api-access-jnks5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327352 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327105 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.327523 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327495 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:54:32.327687 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.327665 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token" pod "eb7e9ee8-8c83-47b5-b501-d71c503fca13" (UID: "eb7e9ee8-8c83-47b5-b501-d71c503fca13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:32.399801 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.399749 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" event={"ID":"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4","Type":"ContainerStarted","Data":"7c013c08d1d6e517edd6db24723ca087fe1b9008ce7105388289bc574fd07ac5"} Nov 03 16:54:32.400032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.399825 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:32.400032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.399849 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" event={"ID":"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4","Type":"ContainerStarted","Data":"0cbdb9d07637b34014fa587e6681edaab70e8176271aa94f770cbad3ded19010"} Nov 03 16:54:32.401735 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.401705 6001 generic.go:356] "Generic (PLEG): container finished" podID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerID="0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3" exitCode=0 Nov 03 16:54:32.401896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.401764 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" Nov 03 16:54:32.401896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.401784 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerDied","Data":"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3"} Nov 03 16:54:32.401896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.401839 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf" event={"ID":"eb7e9ee8-8c83-47b5-b501-d71c503fca13","Type":"ContainerDied","Data":"f5b17537c9ae502c4b1a12a6b9e290e7dc6d0d03819394652ea926a26c6bf62c"} Nov 03 16:54:32.401896 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.401862 6001 scope.go:122] "RemoveContainer" containerID="0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3" Nov 03 16:54:32.418717 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.418679 6001 scope.go:122] "RemoveContainer" containerID="5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f" Nov 03 16:54:32.429133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429102 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.429133 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429131 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7e9ee8-8c83-47b5-b501-d71c503fca13-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:32.429287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429161 6001 scope.go:122] "RemoveContainer" containerID="0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3" Nov 03 16:54:32.429566 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:54:32.429538 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3\": container with ID starting with 0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3 not found: ID does not exist" containerID="0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3" Nov 03 16:54:32.429610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429579 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3"} err="failed to get container status \"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3\": rpc error: code = NotFound desc = could not find container \"0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3\": container with ID starting with 0ae2889602f4fd12e401708bf913a6ef244dcc2a3b852f9df82b7366b7bed7f3 not found: ID does not exist" Nov 03 16:54:32.429610 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429604 6001 scope.go:122] "RemoveContainer" containerID="5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f" Nov 03 16:54:32.429899 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:54:32.429880 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f\": container with ID starting with 5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f not found: ID does not exist" containerID="5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f" Nov 03 16:54:32.429952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.429905 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f"} err="failed to get container status \"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f\": rpc error: code = NotFound desc = could not find container \"5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f\": container with ID starting with 5904d0c2507609bbc720853374d4edbc3e836be8d0629818ea3a68b332bbf60f not found: ID does not exist" Nov 03 16:54:32.446823 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.446755 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" podStartSLOduration=1.44673312 podStartE2EDuration="1.44673312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:32.442076904 +0000 UTC m=+2367.775290473" watchObservedRunningTime="2026-11-03 16:54:32.44673312 +0000 UTC m=+2367.779946691" Nov 03 16:54:32.491348 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.491308 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:54:32.495382 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.495358 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-be8495-openshift-default-68998d7b9-kp7gf"] Nov 03 16:54:32.791503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:32.791407 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" path="/var/lib/kubelet/pods/eb7e9ee8-8c83-47b5-b501-d71c503fca13/volumes" Nov 03 16:54:34.552254 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:34.552220 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:54:39.553689 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.553410 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" podUID="6c12a8e8-3084-467a-b8d7-dac939d1447d" containerName="payload-processing" containerID="cri-o://c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091" gracePeriod=25 Nov 03 16:54:39.859226 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.859199 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:54:39.981001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.980969 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\" (UniqueName: \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\") pod \"6c12a8e8-3084-467a-b8d7-dac939d1447d\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " Nov 03 16:54:39.981177 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.981141 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\") pod \"6c12a8e8-3084-467a-b8d7-dac939d1447d\" (UID: \"6c12a8e8-3084-467a-b8d7-dac939d1447d\") " Nov 03 16:54:39.981544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.981516 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume" pod "6c12a8e8-3084-467a-b8d7-dac939d1447d" (UID: "6c12a8e8-3084-467a-b8d7-dac939d1447d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:39.981709 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.981695 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c12a8e8-3084-467a-b8d7-dac939d1447d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:39.983121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:39.983102 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr" pod "6c12a8e8-3084-467a-b8d7-dac939d1447d" (UID: "6c12a8e8-3084-467a-b8d7-dac939d1447d"). InnerVolumeSpecName "kube-api-access-llstr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:40.083427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.083335 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llstr\" (UniqueName: \"kubernetes.io/projected/6c12a8e8-3084-467a-b8d7-dac939d1447d-kube-api-access-llstr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:54:40.479049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.479013 6001 generic.go:356] "Generic (PLEG): container finished" podID="6c12a8e8-3084-467a-b8d7-dac939d1447d" containerID="c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091" exitCode=0 Nov 03 16:54:40.479231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.479060 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" Nov 03 16:54:40.479231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.479098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" event={"ID":"6c12a8e8-3084-467a-b8d7-dac939d1447d","Type":"ContainerDied","Data":"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091"} Nov 03 16:54:40.479231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.479155 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt" event={"ID":"6c12a8e8-3084-467a-b8d7-dac939d1447d","Type":"ContainerDied","Data":"57be76651e9b3f15ea19bcd8515d840b032e488e4011463f376c6409e242ab53"} Nov 03 16:54:40.479231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.479183 6001 scope.go:122] "RemoveContainer" containerID="c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091" Nov 03 16:54:40.490165 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.490140 6001 scope.go:122] "RemoveContainer" containerID="c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091" Nov 03 16:54:40.490492 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:54:40.490468 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091\": container with ID starting with c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091 not found: ID does not exist" containerID="c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091" Nov 03 16:54:40.490608 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.490499 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091"} err="failed to get container status \"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091\": rpc error: code = NotFound desc = could not find container \"c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091\": container with ID starting with c00b43d64c6d07e1c2fb7f8e62e409d79490a2619d319b5cdb48c1870c57e091 not found: ID does not exist" Nov 03 16:54:40.535333 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.535292 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:54:40.548159 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.548124 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a11d94-5cb6c7ff45-zrgwt"] Nov 03 16:54:40.791391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:40.791310 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c12a8e8-3084-467a-b8d7-dac939d1447d" path="/var/lib/kubelet/pods/6c12a8e8-3084-467a-b8d7-dac939d1447d/volumes" Nov 03 16:54:43.425690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:43.425623 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 16:54:44.106788 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.106743 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 16:54:44.107984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.107866 6001 cpu_manager.go:414] "containerMap: removing container" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:44.108216 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108185 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:44.108349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108250 6001 cpu_manager.go:414] "containerMap: removing container" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:44.108349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108261 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:44.108349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108287 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6c12a8e8-3084-467a-b8d7-dac939d1447d" containerName="payload-processing" Nov 03 16:54:44.108349 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108296 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c12a8e8-3084-467a-b8d7-dac939d1447d" containerName="payload-processing" Nov 03 16:54:44.108677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108582 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:44.108677 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.108604 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c12a8e8-3084-467a-b8d7-dac939d1447d" containerName="payload-processing" Nov 03 16:54:44.115078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.115052 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.138829 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.138776 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-2b49df-openshift-default-dockercfg-dqkdh\"" Nov 03 16:54:44.139948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.139912 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 16:54:44.146008 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.145969 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.146357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.146330 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rwr\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.146766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.146741 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.148059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.148035 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.148326 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.148306 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.148599 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.148577 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.149956 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.149930 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.150207 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.150187 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.150485 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.150463 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257302 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257387 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257463 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rwr\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257608 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257638 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257706 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.257798 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257792 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.258530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257852 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.258530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.257884 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.280475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.258890 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.280475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.259981 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.280475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.260589 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.280475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.260858 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.280475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.261111 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.283477 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.282571 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.292427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.292352 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rwr\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.294529 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.294495 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.295124 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.295073 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\") pod \"e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:44.448835 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:44.448735 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:45.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.136233 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 16:54:45.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.139892 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:45.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.139992 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:45.140503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.140042 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:45.576554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.575485 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerStarted","Data":"e75a3635e62f272c3399906098a6c63712260b05274e4d24c51d5264fd6bb34a"} Nov 03 16:54:45.576554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.575559 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerStarted","Data":"a14397b0e15ab0eecac3fa57c547184d7d0f0c49e30278c38cf6cf3bec5950b7"} Nov 03 16:54:45.634353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:45.634271 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podStartSLOduration=1.63424764 podStartE2EDuration="1.63424764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:45.629048132 +0000 UTC m=+2380.962261700" watchObservedRunningTime="2026-11-03 16:54:45.63424764 +0000 UTC m=+2380.967461207" Nov 03 16:54:46.450670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:46.450636 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:46.456512 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:46.456481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:46.582881 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:46.582849 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:46.584031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:46.584008 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:54:50.905282 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.905247 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 16:54:50.906059 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.906043 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7e9ee8-8c83-47b5-b501-d71c503fca13" containerName="istio-proxy" Nov 03 16:54:50.910187 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.910165 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:50.939212 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.939169 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2b49df-dockercfg-f54zb\"" Nov 03 16:54:50.939401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.939211 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2b49df\"" Nov 03 16:54:50.945825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:50.945790 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 16:54:51.013793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.013735 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbscm\" (UniqueName: \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.021041 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.020981 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.128726 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.128684 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.147467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.129151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbscm\" (UniqueName: \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.156501 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.149435 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.199279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.199234 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 16:54:51.204767 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.204731 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.204974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.204909 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 16:54:51.218765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.218730 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbscm\" (UniqueName: \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\") pod \"payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.225982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.225932 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.350871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.350830 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzrw\" (UniqueName: \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.351066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.351052 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.453182 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.453076 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.453362 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.453259 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzrw\" (UniqueName: \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.453893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.453873 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.477707 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.477674 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzrw\" (UniqueName: \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\") pod \"payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.520995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.520960 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:51.535556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.535521 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 16:54:51.535930 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:51.535903 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0014c4_3df4_4657_8350_9db0c4ad78c6.slice/crio-1539b1048b86d202d658232810f6c1d912a973d447931b2266488c1a8f776516 WatchSource:0}: Error finding container 1539b1048b86d202d658232810f6c1d912a973d447931b2266488c1a8f776516: Status 404 returned error can't find the container with id 1539b1048b86d202d658232810f6c1d912a973d447931b2266488c1a8f776516 Nov 03 16:54:51.677594 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.677555 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" event={"ID":"8e0014c4-3df4-4657-8350-9db0c4ad78c6","Type":"ContainerStarted","Data":"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3"} Nov 03 16:54:51.677782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.677615 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" event={"ID":"8e0014c4-3df4-4657-8350-9db0c4ad78c6","Type":"ContainerStarted","Data":"1539b1048b86d202d658232810f6c1d912a973d447931b2266488c1a8f776516"} Nov 03 16:54:51.677782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.677639 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:54:51.763470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.743399 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" podStartSLOduration=1.743376813 podStartE2EDuration="1.743376813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:51.741034667 +0000 UTC m=+2387.074248241" watchObservedRunningTime="2026-11-03 16:54:51.743376813 +0000 UTC m=+2387.076590383" Nov 03 16:54:51.830803 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:51.830769 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bdf97b_8183_4878_ac24_9c88fd170cdb.slice/crio-7459834ba0e09aedd045b12e6847d6c9860b00a02de521a7653950b231b8e68a WatchSource:0}: Error finding container 7459834ba0e09aedd045b12e6847d6c9860b00a02de521a7653950b231b8e68a: Status 404 returned error can't find the container with id 7459834ba0e09aedd045b12e6847d6c9860b00a02de521a7653950b231b8e68a Nov 03 16:54:51.831749 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:51.831722 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 16:54:52.253657 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:54:52.253618 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:54:52.687209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:52.687094 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" event={"ID":"a6bdf97b-8183-4878-ac24-9c88fd170cdb","Type":"ContainerStarted","Data":"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243"} Nov 03 16:54:52.687209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:52.687169 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" event={"ID":"a6bdf97b-8183-4878-ac24-9c88fd170cdb","Type":"ContainerStarted","Data":"7459834ba0e09aedd045b12e6847d6c9860b00a02de521a7653950b231b8e68a"} Nov 03 16:54:52.687209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:52.687204 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:54:56.733952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.733894 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" podStartSLOduration=6.733878505 podStartE2EDuration="6.733878505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:52.74255917 +0000 UTC m=+2388.075772739" watchObservedRunningTime="2026-11-03 16:54:56.733878505 +0000 UTC m=+2392.067092122" Nov 03 16:54:56.737336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.737309 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 16:54:56.742295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.742270 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.767627 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.767592 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 16:54:56.773201 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.773172 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-c6f303-openshift-default-dockercfg-jfcgv\"" Nov 03 16:54:56.844410 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844361 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844421 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844619 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844670 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844826 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844761 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844890 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844873 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.844951 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.844913 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.845013 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.845001 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.845073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.845046 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqfn\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947583 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947646 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqfn\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947778 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947815 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947905 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947947 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.947980 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.948071 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.948124 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.949042 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.949338 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.950108 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.968948 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.950349 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.969825 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.969342 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.972419 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.972380 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.985521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.981476 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqfn\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:56.987295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:56.987263 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:57.021175 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.021130 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\") pod \"e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:57.060285 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.060215 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:57.391824 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:54:57.391787 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957bd4e6_a2de_4b74_94c7_c73e21c1534e.slice/crio-53729e57d0319ce6910cd3f4968630d404c7e74a0f1a2985af6dd5e5aa4917fb WatchSource:0}: Error finding container 53729e57d0319ce6910cd3f4968630d404c7e74a0f1a2985af6dd5e5aa4917fb: Status 404 returned error can't find the container with id 53729e57d0319ce6910cd3f4968630d404c7e74a0f1a2985af6dd5e5aa4917fb Nov 03 16:54:57.394141 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.394113 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 16:54:57.394858 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.394826 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:57.394937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.394893 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:57.394937 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.394923 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:54:57.734974 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.734937 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerStarted","Data":"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661"} Nov 03 16:54:57.735574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.734999 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerStarted","Data":"53729e57d0319ce6910cd3f4968630d404c7e74a0f1a2985af6dd5e5aa4917fb"} Nov 03 16:54:57.821730 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:57.821630 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podStartSLOduration=1.821606606 podStartE2EDuration="1.821606606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:57.781498985 +0000 UTC m=+2393.114712555" watchObservedRunningTime="2026-11-03 16:54:57.821606606 +0000 UTC m=+2393.154820176" Nov 03 16:54:58.060522 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:58.060380 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:58.067777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:58.067730 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:58.743181 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:58.743148 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:54:58.744246 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:54:58.744223 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:55:02.251783 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.251741 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 16:55:02.256531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.256502 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.277601 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.277562 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 16:55:02.284921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.284890 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c6f303\"" Nov 03 16:55:02.285146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.284989 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c6f303-dockercfg-gtqb2\"" Nov 03 16:55:02.391746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.381639 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.391746 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.381711 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh77g\" (UniqueName: \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.519179 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.519055 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.519544 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.519508 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh77g\" (UniqueName: \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.528407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.520328 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.551901 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.551790 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh77g\" (UniqueName: \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\") pod \"payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.575469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.574732 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:02.690402 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.690371 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 16:55:02.885528 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:02.885481 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfafb4ea_c764_48d7_9f9f_f7a458f662ed.slice/crio-06e8409555b282c93f99932ea8790d4001d2514af02d8e4bcdab63865471bb51 WatchSource:0}: Error finding container 06e8409555b282c93f99932ea8790d4001d2514af02d8e4bcdab63865471bb51: Status 404 returned error can't find the container with id 06e8409555b282c93f99932ea8790d4001d2514af02d8e4bcdab63865471bb51 Nov 03 16:55:02.887005 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:02.886980 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 16:55:03.699295 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:03.699264 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 16:55:03.804996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:03.804958 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" event={"ID":"bfafb4ea-c764-48d7-9f9f-f7a458f662ed","Type":"ContainerStarted","Data":"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783"} Nov 03 16:55:03.805159 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:03.805014 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" event={"ID":"bfafb4ea-c764-48d7-9f9f-f7a458f662ed","Type":"ContainerStarted","Data":"06e8409555b282c93f99932ea8790d4001d2514af02d8e4bcdab63865471bb51"} Nov 03 16:55:03.805159 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:03.805099 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:03.851887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:03.850409 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" podStartSLOduration=1.8503886280000001 podStartE2EDuration="1.850388628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:03.847112451 +0000 UTC m=+2399.180326019" watchObservedRunningTime="2026-11-03 16:55:03.850388628 +0000 UTC m=+2399.183602195" Nov 03 16:55:05.175921 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:05.175891 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:55:05.183657 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:05.183630 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 16:55:08.425421 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.425368 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 16:55:08.455192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.454635 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.461292 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.461258 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 16:55:08.481673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.481637 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-001802-openshift-default-dockercfg-skw4x\"" Nov 03 16:55:08.583319 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583279 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583473 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583516 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583545 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583617 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583755 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583728 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583777 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6hh\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.583931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.583909 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698682 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698722 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698800 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698845 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698903 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6hh\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.698935 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.699033 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.699701 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.699962 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.702813 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.703046 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.708545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.704688 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.723071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.723039 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.738467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.737732 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.739887 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.739856 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.748336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.748299 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6hh\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\") pod \"e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:08.777069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:08.775752 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:09.399301 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:09.399256 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86121e97_1bf0_4aaa_afc8_1474a7b6fd64.slice/crio-6b868484dc3ac2b9fd23a24360569d7cd00d1b24b7e1bb4dd5b6a0bdc2312b0a WatchSource:0}: Error finding container 6b868484dc3ac2b9fd23a24360569d7cd00d1b24b7e1bb4dd5b6a0bdc2312b0a: Status 404 returned error can't find the container with id 6b868484dc3ac2b9fd23a24360569d7cd00d1b24b7e1bb4dd5b6a0bdc2312b0a Nov 03 16:55:09.399888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.399834 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 16:55:09.402393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.402350 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:09.402506 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.402469 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:09.402554 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.402524 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:09.903562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.903521 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerStarted","Data":"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813"} Nov 03 16:55:09.904034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.903585 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerStarted","Data":"6b868484dc3ac2b9fd23a24360569d7cd00d1b24b7e1bb4dd5b6a0bdc2312b0a"} Nov 03 16:55:09.972061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:09.972000 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podStartSLOduration=1.971982341 podStartE2EDuration="1.971982341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:09.969241496 +0000 UTC m=+2405.302455070" watchObservedRunningTime="2026-11-03 16:55:09.971982341 +0000 UTC m=+2405.305195909" Nov 03 16:55:10.777243 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:10.777204 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:10.791139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:10.791109 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:10.922464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:10.922052 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:10.923232 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:10.923206 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:55:14.481457 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.480050 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 16:55:14.488087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.488060 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.508093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.508055 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 16:55:14.514436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.514400 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-001802-dockercfg-q6dtk\"" Nov 03 16:55:14.514649 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.514491 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-001802\"" Nov 03 16:55:14.525719 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.525688 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 16:55:14.531914 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.531886 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.576792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.576732 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wlgp\" (UniqueName: \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.577000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.576890 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.577000 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.576954 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46lb\" (UniqueName: \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.578894 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.577229 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.582355 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.582324 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 16:55:14.679239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.679194 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wlgp\" (UniqueName: \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.679430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.679329 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.679430 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.679378 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46lb\" (UniqueName: \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.679683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.679658 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.680517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.680490 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.680807 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.680782 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.710420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.710381 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46lb\" (UniqueName: \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\") pod \"payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.712494 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.712462 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wlgp\" (UniqueName: \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\") pod \"payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.865525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.852974 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:14.890997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.889669 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:14.981521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:14.979865 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 16:55:15.420357 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:15.420312 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6304a6_8964_4f43_b35a_0a98edbcf648.slice/crio-7287eae06a434f9bfd63a22d5ec75b2d2eeec10f096aea178caa126fc52b056c WatchSource:0}: Error finding container 7287eae06a434f9bfd63a22d5ec75b2d2eeec10f096aea178caa126fc52b056c: Status 404 returned error can't find the container with id 7287eae06a434f9bfd63a22d5ec75b2d2eeec10f096aea178caa126fc52b056c Nov 03 16:55:15.421215 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:15.421182 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 16:55:15.439842 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:15.439794 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0db1692_459c_4b2c_8c20_440f391dadd7.slice/crio-b22ce30c4c7bd6ba363a6c3c95983cc18d4ee7162cb5297263022e770ff9ab61 WatchSource:0}: Error finding container b22ce30c4c7bd6ba363a6c3c95983cc18d4ee7162cb5297263022e770ff9ab61: Status 404 returned error can't find the container with id b22ce30c4c7bd6ba363a6c3c95983cc18d4ee7162cb5297263022e770ff9ab61 Nov 03 16:55:15.440851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:15.440824 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 16:55:16.070393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.070339 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" event={"ID":"da6304a6-8964-4f43-b35a-0a98edbcf648","Type":"ContainerStarted","Data":"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae"} Nov 03 16:55:16.070927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.070410 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" event={"ID":"da6304a6-8964-4f43-b35a-0a98edbcf648","Type":"ContainerStarted","Data":"7287eae06a434f9bfd63a22d5ec75b2d2eeec10f096aea178caa126fc52b056c"} Nov 03 16:55:16.070927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.070466 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:16.072585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.072551 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" event={"ID":"f0db1692-459c-4b2c-8c20-440f391dadd7","Type":"ContainerStarted","Data":"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2"} Nov 03 16:55:16.072737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.072603 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" event={"ID":"f0db1692-459c-4b2c-8c20-440f391dadd7","Type":"ContainerStarted","Data":"b22ce30c4c7bd6ba363a6c3c95983cc18d4ee7162cb5297263022e770ff9ab61"} Nov 03 16:55:16.072737 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.072624 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:16.117280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.117213 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" podStartSLOduration=2.11719887 podStartE2EDuration="2.11719887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:16.113232832 +0000 UTC m=+2411.446446405" watchObservedRunningTime="2026-11-03 16:55:16.11719887 +0000 UTC m=+2411.450412438" Nov 03 16:55:16.158465 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:16.158397 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" podStartSLOduration=2.158384252 podStartE2EDuration="2.158384252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:16.157831098 +0000 UTC m=+2411.491044669" watchObservedRunningTime="2026-11-03 16:55:16.158384252 +0000 UTC m=+2411.491597816" Nov 03 16:55:19.044002 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.043969 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:55:19.050194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.050166 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.069514 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.069482 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:55:19.076790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.076755 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-aaea00\"" Nov 03 16:55:19.076790 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.076788 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-aaea00-dockercfg-xvckl\"" Nov 03 16:55:19.352912 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.352811 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzckt\" (UniqueName: \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.353127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.352937 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.356587 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.356553 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:55:19.370461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.367654 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:55:19.383464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.383008 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.454436 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.454392 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.454652 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.454577 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclbk\" (UniqueName: \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.454722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.454703 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzckt\" (UniqueName: \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.454783 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.454771 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.455571 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.455548 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.482781 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.482741 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzckt\" (UniqueName: \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\") pod \"payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.562977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.561576 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclbk\" (UniqueName: \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.562977 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.561918 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.563309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.562982 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.674574 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.673276 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:19.709038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.709005 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclbk\" (UniqueName: \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\") pod \"payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:19.716925 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:19.716525 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:20.020968 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:20.020939 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d73940_4449_41c8_b3c2_1a4871bfa8a5.slice/crio-39f373f32d1445b572c39f34b5f89b54617cff76022ade2b9b2627b48bab69f5 WatchSource:0}: Error finding container 39f373f32d1445b572c39f34b5f89b54617cff76022ade2b9b2627b48bab69f5: Status 404 returned error can't find the container with id 39f373f32d1445b572c39f34b5f89b54617cff76022ade2b9b2627b48bab69f5 Nov 03 16:55:20.023899 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.023873 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:55:20.067223 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:55:20.067181 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15877f2_821c_4885_9336_b0f7952b9440.slice/crio-3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0 WatchSource:0}: Error finding container 3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0: Status 404 returned error can't find the container with id 3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0 Nov 03 16:55:20.069183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.069160 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:55:20.138384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.138352 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" event={"ID":"55d73940-4449-41c8-b3c2-1a4871bfa8a5","Type":"ContainerStarted","Data":"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8"} Nov 03 16:55:20.138635 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.138425 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" event={"ID":"55d73940-4449-41c8-b3c2-1a4871bfa8a5","Type":"ContainerStarted","Data":"39f373f32d1445b572c39f34b5f89b54617cff76022ade2b9b2627b48bab69f5"} Nov 03 16:55:20.138635 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.138501 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:20.142072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.142030 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" event={"ID":"e15877f2-821c-4885-9336-b0f7952b9440","Type":"ContainerStarted","Data":"5adc394badd20d85b23977de2e8b2593d98a72b74a254eaec251aceb9f6dd46e"} Nov 03 16:55:20.142193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.142104 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" event={"ID":"e15877f2-821c-4885-9336-b0f7952b9440","Type":"ContainerStarted","Data":"3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0"} Nov 03 16:55:20.184849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:20.184793 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" podStartSLOduration=1.184777429 podStartE2EDuration="1.184777429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:20.182317566 +0000 UTC m=+2415.515531137" watchObservedRunningTime="2026-11-03 16:55:20.184777429 +0000 UTC m=+2415.517990995" Nov 03 16:55:21.156190 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.156154 6001 generic.go:356] "Generic (PLEG): container finished" podID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerID="f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a" exitCode=0 Nov 03 16:55:21.156672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.156221 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerDied","Data":"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a"} Nov 03 16:55:21.156672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.156500 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:21.156822 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.156744 6001 scope.go:122] "RemoveContainer" containerID="f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a" Nov 03 16:55:21.158408 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.158375 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:21.158558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.158497 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:21.158558 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.158539 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:21.201305 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.201257 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" podStartSLOduration=2.201243017 podStartE2EDuration="2.201243017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:21.200400178 +0000 UTC m=+2416.533613747" watchObservedRunningTime="2026-11-03 16:55:21.201243017 +0000 UTC m=+2416.534456585" Nov 03 16:55:21.408521 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.408403 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 16:55:21.411885 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.411861 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.431046 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.431013 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-1e9d48\"" Nov 03 16:55:21.439791 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.439744 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 16:55:21.452863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.452827 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-1e9d48-dockercfg-49ptz\"" Nov 03 16:55:21.500301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.500262 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.500538 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.500514 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzcq\" (UniqueName: \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.605469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.604545 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzcq\" (UniqueName: \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.605469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.604812 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.609464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.605904 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.697298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.697205 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzcq\" (UniqueName: \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\") pod \"payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:21.732869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:21.732565 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:22.099183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.099142 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 16:55:22.193107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.193064 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" event={"ID":"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a","Type":"ContainerStarted","Data":"edf052385f7287598e0d0226471acf112671069f08b338e3e780ef41059d2407"} Nov 03 16:55:22.195469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.195453 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerStarted","Data":"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39"} Nov 03 16:55:22.278796 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:55:22.278751 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:55:22.737568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.736971 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:55:22.738777 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.738744 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 16:55:22.738964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:22.738803 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 16:55:23.205630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:23.205542 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" event={"ID":"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a","Type":"ContainerStarted","Data":"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb"} Nov 03 16:55:23.206031 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:23.205652 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:23.273161 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:23.273084 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" podStartSLOduration=2.273063661 podStartE2EDuration="2.273063661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:23.269840243 +0000 UTC m=+2418.603053813" watchObservedRunningTime="2026-11-03 16:55:23.273063661 +0000 UTC m=+2418.606277231" Nov 03 16:55:23.736017 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:23.735982 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 16:55:23.736217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:23.736036 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 16:55:24.735927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:24.735894 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 16:55:24.736381 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:24.735943 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 16:55:25.735198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:25.735158 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 16:55:25.735405 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:25.735214 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 16:55:26.368518 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:26.366552 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:55:26.771766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:26.771729 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Nov 03 16:55:26.771995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:26.771801 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Nov 03 16:55:27.084078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:27.083994 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 16:55:27.084078 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:27.084069 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 16:55:27.740214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:27.740180 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:55:27.741269 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:27.741247 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 16:55:31.169837 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:31.169808 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:55:32.199290 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:32.199259 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:55:34.217123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:34.217090 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 16:55:39.379724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.379625 6001 generic.go:356] "Generic (PLEG): container finished" podID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerID="e75a3635e62f272c3399906098a6c63712260b05274e4d24c51d5264fd6bb34a" exitCode=0 Nov 03 16:55:39.379724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.379698 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerDied","Data":"e75a3635e62f272c3399906098a6c63712260b05274e4d24c51d5264fd6bb34a"} Nov 03 16:55:39.380233 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.380218 6001 scope.go:122] "RemoveContainer" containerID="e75a3635e62f272c3399906098a6c63712260b05274e4d24c51d5264fd6bb34a" Nov 03 16:55:39.382153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.382124 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:39.382218 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.382206 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:39.382263 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:39.382250 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:40.390989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:40.390947 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerStarted","Data":"e8b41ecbb320ce9cce3779c915dda4d7aa548a58cd711c7e6474df81186f7061"} Nov 03 16:55:40.460351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:40.459859 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:55:40.460351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:40.460023 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:40.460351 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:40.460085 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:41.452468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:41.450748 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:41.452468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:41.450811 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:42.455020 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:42.454983 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:42.455673 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:42.455638 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:43.454891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:43.454850 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:43.455198 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:43.455166 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:44.457782 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:44.457739 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:44.489372 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:44.489334 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:45.458603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:45.458558 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body= Nov 03 16:55:45.459294 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:45.459245 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" Nov 03 16:55:46.465784 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:46.465700 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:55:46.467838 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:46.467818 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:55:46.470044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:46.470021 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 16:55:52.254189 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:55:52.254141 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:55:52.573152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.573048 6001 generic.go:356] "Generic (PLEG): container finished" podID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerID="897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661" exitCode=0 Nov 03 16:55:52.573152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.573125 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerDied","Data":"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661"} Nov 03 16:55:52.573785 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.573757 6001 scope.go:122] "RemoveContainer" containerID="897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661" Nov 03 16:55:52.576617 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.576576 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:52.576756 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.576735 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:52.576860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:52.576843 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:55:53.583401 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:53.583359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerStarted","Data":"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b"} Nov 03 16:55:54.062891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:54.062849 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:55:54.063303 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:54.063273 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:54.063503 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:54.063476 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:55:55.061845 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:55.061808 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:55.062491 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:55.062435 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:55:56.064903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:56.064864 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:56.065622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:56.065585 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:55:57.063911 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:57.063871 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:57.065238 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:57.065186 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:55:58.066473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:58.064947 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:58.066473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:58.065014 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:55:58.752632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:58.752590 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:55:59.064534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:59.063873 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body= Nov 03 16:55:59.064534 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:55:59.063957 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" Nov 03 16:56:00.074504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:00.074463 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:56:00.076398 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:00.076371 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 16:56:07.734634 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.734572 6001 generic.go:356] "Generic (PLEG): container finished" podID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerID="9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813" exitCode=0 Nov 03 16:56:07.735142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.734643 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerDied","Data":"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813"} Nov 03 16:56:07.735142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.735126 6001 scope.go:122] "RemoveContainer" containerID="9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813" Nov 03 16:56:07.736964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.736935 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:07.737027 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.737013 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:07.737069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:07.737047 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:08.756683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:08.756634 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerStarted","Data":"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db"} Nov 03 16:56:08.780688 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:08.780657 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:56:08.783684 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:08.783651 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:08.783827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:08.783701 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:09.776648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:09.776612 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:09.777087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:09.776675 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:10.776742 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:10.776708 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:10.777192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:10.776767 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:10.922566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:10.922530 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:56:11.776741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:11.776702 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:11.777273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:11.776786 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:12.776532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:12.776492 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:12.776741 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:12.776558 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:13.777080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:13.777045 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 16:56:13.777559 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:13.777111 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 16:56:14.781592 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:14.781549 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:56:14.793329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:14.793294 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 16:56:22.254043 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:56:22.253999 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:56:42.592075 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.592028 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:56:42.597926 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.597898 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.616763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.616731 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:56:42.625863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.623345 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"kube-root-ca.crt\"" Nov 03 16:56:42.625863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.623389 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"openshift-service-ca.crt\"" Nov 03 16:56:42.625863 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.624529 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"auth-test-model-5a5ab1-kserve-self-signed-certs\"" Nov 03 16:56:42.626877 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.626853 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5a5ab1\"/\"default-dockercfg-cwbfj\"" Nov 03 16:56:42.669704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.669667 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.669952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.669801 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzf4\" (UniqueName: \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.669952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.669867 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.669952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.669883 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.669952 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.669899 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.670211 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.670165 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.771860 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.771823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.771909 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzf4\" (UniqueName: \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.771949 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.771967 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.772164 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.772384 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772417 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.772411 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772639 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.772528 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.772865 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.772843 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.774433 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.774408 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.774570 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.774550 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.796389 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.796359 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzf4\" (UniqueName: \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\") pod \"auth-test-model-5a5ab1-kserve-78448f77-jz6rw\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:42.923969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:42.923868 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:43.297703 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:43.297613 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:56:44.123044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:44.122999 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" event={"ID":"e87be5c3-4695-4b68-86a4-4fc2b6e152d8","Type":"ContainerStarted","Data":"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed"} Nov 03 16:56:44.123044 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:44.123053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" event={"ID":"e87be5c3-4695-4b68-86a4-4fc2b6e152d8","Type":"ContainerStarted","Data":"11832adba4445a4e8df488934acc92c118bade926a596439cb4bcadc164db495"} Nov 03 16:56:44.173656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:44.173582 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" podStartSLOduration=2.173563229 podStartE2EDuration="2.173563229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:44.170427522 +0000 UTC m=+2499.503641095" watchObservedRunningTime="2026-11-03 16:56:44.173563229 +0000 UTC m=+2499.506776799" Nov 03 16:56:44.929575 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:44.925734 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:46.036347 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:46.036314 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw"] Nov 03 16:56:46.048377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:46.048348 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27a19b17-7npmw"] Nov 03 16:56:46.807464 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:46.805616 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d83d6a7-cfbe-45a8-bcbb-c09692eafd60" path="/var/lib/kubelet/pods/5d83d6a7-cfbe-45a8-bcbb-c09692eafd60/volumes" Nov 03 16:56:51.031009 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:51.030972 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg"] Nov 03 16:56:51.043189 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:51.043156 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90456f76-f9qtg"] Nov 03 16:56:52.006598 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.006553 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 16:56:52.011137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.011118 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.038792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.038759 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.038800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.038892 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.038922 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.038983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8g2q\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.039032 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.039091 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.039177 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.039236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.039206 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.048135 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.048100 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9a80f0-openshift-default-dockercfg-skbfb\"" Nov 03 16:56:52.050672 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.050651 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 16:56:52.140437 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140410 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140483 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140555 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140575 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140635 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140655 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140751 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.140811 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140776 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140844 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8g2q\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141033 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.140962 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141138 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.141082 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141260 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.141238 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141353 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.141333 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.141562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.141539 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.143076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.143054 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.143431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.143412 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.164170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.164145 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.164710 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.164691 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8g2q\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.257161 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:56:52.257064 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:56:52.326497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.326452 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:52.676891 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:56:52.676860 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7266a271_f02f_45bd_befa_b68dc8e2826d.slice/crio-9c1f9c6406e615ae69023af1228984b9b3a84b2bed09fcbe61b93c03993fba3a WatchSource:0}: Error finding container 9c1f9c6406e615ae69023af1228984b9b3a84b2bed09fcbe61b93c03993fba3a: Status 404 returned error can't find the container with id 9c1f9c6406e615ae69023af1228984b9b3a84b2bed09fcbe61b93c03993fba3a Nov 03 16:56:52.679820 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.679777 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:52.679961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.679854 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:52.679961 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.679890 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:56:52.681139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.681103 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 16:56:52.793828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.793787 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58e9589-b19b-459c-b432-47ac6b30abd4" path="/var/lib/kubelet/pods/e58e9589-b19b-459c-b432-47ac6b30abd4/volumes" Nov 03 16:56:52.926048 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.925954 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:52.944230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:52.944197 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:53.209950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.209859 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" event={"ID":"7266a271-f02f-45bd-befa-b68dc8e2826d","Type":"ContainerStarted","Data":"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc"} Nov 03 16:56:53.209950 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.209930 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" event={"ID":"7266a271-f02f-45bd-befa-b68dc8e2826d","Type":"ContainerStarted","Data":"9c1f9c6406e615ae69023af1228984b9b3a84b2bed09fcbe61b93c03993fba3a"} Nov 03 16:56:53.221891 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.221859 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:56:53.262346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.262278 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podStartSLOduration=2.262263168 podStartE2EDuration="2.262263168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:53.25822955 +0000 UTC m=+2508.591443119" watchObservedRunningTime="2026-11-03 16:56:53.262263168 +0000 UTC m=+2508.595476737" Nov 03 16:56:53.332476 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.331135 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:56:53.335543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.334308 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:53.335543 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:53.334379 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:54.326861 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:54.326826 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:54.327304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:54.326885 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:55.327833 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:55.327796 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:55.328230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:55.327855 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:56.032087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:56.031975 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n"] Nov 03 16:56:56.061471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:56.058348 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8b2a94-8km2n"] Nov 03 16:56:56.327831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:56.327751 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:56.327831 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:56.327806 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:56.791695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:56.791662 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140cd1c8-3ee4-4920-8806-021f6ef22535" path="/var/lib/kubelet/pods/140cd1c8-3ee4-4920-8806-021f6ef22535/volumes" Nov 03 16:56:57.327101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:57.327067 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:57.327298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:57.327127 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:58.327188 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:58.327156 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:58.327621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:58.327201 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:56:59.326916 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:59.326875 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" start-of-body= Nov 03 16:56:59.327097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:56:59.326935 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.235:15021/healthz/ready\": dial tcp 10.134.0.235:15021: connect: connection refused" Nov 03 16:57:00.332461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:00.332411 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:57:00.332982 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:00.332854 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:57:00.333551 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:00.333529 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 16:57:14.035996 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:14.035954 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256"] Nov 03 16:57:14.048774 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:14.048738 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32e1621f-nh256"] Nov 03 16:57:14.792938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:14.792904 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28e3b40-7747-4e77-957f-c70e80bbe596" path="/var/lib/kubelet/pods/c28e3b40-7747-4e77-957f-c70e80bbe596/volumes" Nov 03 16:57:22.252879 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:57:22.252796 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:57:27.033106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:27.033071 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57"] Nov 03 16:57:27.045061 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:27.045020 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e9dc695-9xb57"] Nov 03 16:57:28.790931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:28.790890 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c137a8-f7e8-4326-b3eb-9170945a910f" path="/var/lib/kubelet/pods/77c137a8-f7e8-4326-b3eb-9170945a910f/volumes" Nov 03 16:57:35.443888 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:35.443848 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:57:35.477171 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:35.477135 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:57:40.445206 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.445126 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" podUID="e15877f2-821c-4885-9336-b0f7952b9440" containerName="payload-processing" containerID="cri-o://5adc394badd20d85b23977de2e8b2593d98a72b74a254eaec251aceb9f6dd46e" gracePeriod=25 Nov 03 16:57:40.713603 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.712966 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" podUID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" containerName="payload-pre-processing" containerID="cri-o://6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8" gracePeriod=25 Nov 03 16:57:40.756070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.756038 6001 generic.go:356] "Generic (PLEG): container finished" podID="e15877f2-821c-4885-9336-b0f7952b9440" containerID="5adc394badd20d85b23977de2e8b2593d98a72b74a254eaec251aceb9f6dd46e" exitCode=0 Nov 03 16:57:40.756205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.756109 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" event={"ID":"e15877f2-821c-4885-9336-b0f7952b9440","Type":"ContainerDied","Data":"5adc394badd20d85b23977de2e8b2593d98a72b74a254eaec251aceb9f6dd46e"} Nov 03 16:57:40.756205 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.756191 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" event={"ID":"e15877f2-821c-4885-9336-b0f7952b9440","Type":"ContainerDied","Data":"3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0"} Nov 03 16:57:40.756304 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.756210 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bad71a01d59783be8e13fc4c5f3604d0021107a750370f6239a1328c4b3c8e0" Nov 03 16:57:40.799872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.799845 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:57:40.894301 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.894253 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\" (UniqueName: \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\") pod \"e15877f2-821c-4885-9336-b0f7952b9440\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " Nov 03 16:57:40.894539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.894410 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\") pod \"e15877f2-821c-4885-9336-b0f7952b9440\" (UID: \"e15877f2-821c-4885-9336-b0f7952b9440\") " Nov 03 16:57:40.894852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.894816 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume" pod "e15877f2-821c-4885-9336-b0f7952b9440" (UID: "e15877f2-821c-4885-9336-b0f7952b9440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:57:40.895766 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.895741 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15877f2-821c-4885-9336-b0f7952b9440-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:40.900006 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.899975 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk" pod "e15877f2-821c-4885-9336-b0f7952b9440" (UID: "e15877f2-821c-4885-9336-b0f7952b9440"). InnerVolumeSpecName "kube-api-access-rclbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:40.997016 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.996976 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclbk\" (UniqueName: \"kubernetes.io/projected/e15877f2-821c-4885-9336-b0f7952b9440-kube-api-access-rclbk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:40.999550 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:40.999517 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:57:41.097879 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.097837 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\" (UniqueName: \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\") pod \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " Nov 03 16:57:41.098137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.098121 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\") pod \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\" (UID: \"55d73940-4449-41c8-b3c2-1a4871bfa8a5\") " Nov 03 16:57:41.098622 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.098591 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume" pod "55d73940-4449-41c8-b3c2-1a4871bfa8a5" (UID: "55d73940-4449-41c8-b3c2-1a4871bfa8a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:57:41.098745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.098717 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55d73940-4449-41c8-b3c2-1a4871bfa8a5-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:41.100276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.100246 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt" pod "55d73940-4449-41c8-b3c2-1a4871bfa8a5" (UID: "55d73940-4449-41c8-b3c2-1a4871bfa8a5"). InnerVolumeSpecName "kube-api-access-kzckt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:41.200152 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.200113 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzckt\" (UniqueName: \"kubernetes.io/projected/55d73940-4449-41c8-b3c2-1a4871bfa8a5-kube-api-access-kzckt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:41.293678 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.293642 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:57:41.765120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765081 6001 generic.go:356] "Generic (PLEG): container finished" podID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" containerID="6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8" exitCode=0 Nov 03 16:57:41.765623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765128 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" Nov 03 16:57:41.765623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765171 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" event={"ID":"55d73940-4449-41c8-b3c2-1a4871bfa8a5","Type":"ContainerDied","Data":"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8"} Nov 03 16:57:41.765623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765228 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6" event={"ID":"55d73940-4449-41c8-b3c2-1a4871bfa8a5","Type":"ContainerDied","Data":"39f373f32d1445b572c39f34b5f89b54617cff76022ade2b9b2627b48bab69f5"} Nov 03 16:57:41.765623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765256 6001 scope.go:122] "RemoveContainer" containerID="6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8" Nov 03 16:57:41.765623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.765387 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z" Nov 03 16:57:41.777669 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.777584 6001 scope.go:122] "RemoveContainer" containerID="6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8" Nov 03 16:57:41.778155 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:57:41.778133 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8\": container with ID starting with 6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8 not found: ID does not exist" containerID="6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8" Nov 03 16:57:41.778259 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.778167 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8"} err="failed to get container status \"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8\": rpc error: code = NotFound desc = could not find container \"6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8\": container with ID starting with 6fd77ec9dfc68678eb972b85b87070c85edceb75d1b1a7083d9c0dc9cba36be8 not found: ID does not exist" Nov 03 16:57:41.841810 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.841773 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:57:41.854091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.854058 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-aaea00-85db486bcf-j85z6"] Nov 03 16:57:41.889612 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.889573 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:57:41.895038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:41.895005 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-aaea00-75fc744b89-xwf8z"] Nov 03 16:57:42.031504 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:42.031391 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm"] Nov 03 16:57:42.035964 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:42.035933 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33243299-h5fhm"] Nov 03 16:57:42.791386 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:42.791352 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" path="/var/lib/kubelet/pods/55d73940-4449-41c8-b3c2-1a4871bfa8a5/volumes" Nov 03 16:57:42.791938 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:42.791918 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc50cff-43a3-4bff-8fb0-b0b648f74427" path="/var/lib/kubelet/pods/9bc50cff-43a3-4bff-8fb0-b0b648f74427/volumes" Nov 03 16:57:42.792350 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:42.792334 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15877f2-821c-4885-9336-b0f7952b9440" path="/var/lib/kubelet/pods/e15877f2-821c-4885-9336-b0f7952b9440/volumes" Nov 03 16:57:49.000792 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.000755 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 16:57:49.001573 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001556 6001 cpu_manager.go:414] "containerMap: removing container" podUID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" containerName="payload-pre-processing" Nov 03 16:57:49.001629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001574 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" containerName="payload-pre-processing" Nov 03 16:57:49.001629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001584 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e15877f2-821c-4885-9336-b0f7952b9440" containerName="payload-processing" Nov 03 16:57:49.001629 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001589 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15877f2-821c-4885-9336-b0f7952b9440" containerName="payload-processing" Nov 03 16:57:49.001744 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001734 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d73940-4449-41c8-b3c2-1a4871bfa8a5" containerName="payload-pre-processing" Nov 03 16:57:49.001780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.001745 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15877f2-821c-4885-9336-b0f7952b9440" containerName="payload-processing" Nov 03 16:57:49.007082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.007055 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.030192 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.030155 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 16:57:49.036980 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.036941 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c1837e-openshift-default-dockercfg-7ldp7\"" Nov 03 16:57:49.199480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199185 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199270 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199348 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199480 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199375 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199494 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199524 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199575 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199647 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhr8\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.199875 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.199792 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.304943 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.304998 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305053 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhr8\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305284 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305276 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305316 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305459 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305374 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305538 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.305585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.305568 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.306745 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.306719 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.307660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.307634 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.308296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.308274 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.309091 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.309058 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.309407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.309381 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.311119 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.311103 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.317296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.317276 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.331080 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.331050 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhr8\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.331486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.331464 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\") pod \"e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.622927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.622844 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:49.944397 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:57:49.944356 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244a6ce2_1c95_4b87_a843_32fc0a178838.slice/crio-13c8f3b7fc602fec37f3b8f447d367ad24b16a8b68dd6c594f6425b9f2a1ca88 WatchSource:0}: Error finding container 13c8f3b7fc602fec37f3b8f447d367ad24b16a8b68dd6c594f6425b9f2a1ca88: Status 404 returned error can't find the container with id 13c8f3b7fc602fec37f3b8f447d367ad24b16a8b68dd6c594f6425b9f2a1ca88 Nov 03 16:57:49.946099 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.945997 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 16:57:49.947134 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.947104 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:57:49.947193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.947164 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:57:49.947230 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:49.947193 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:57:50.885416 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:50.885380 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerStarted","Data":"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9"} Nov 03 16:57:50.885870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:50.885472 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerStarted","Data":"13c8f3b7fc602fec37f3b8f447d367ad24b16a8b68dd6c594f6425b9f2a1ca88"} Nov 03 16:57:50.959966 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:50.959910 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podStartSLOduration=2.959894877 podStartE2EDuration="2.959894877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:50.945966271 +0000 UTC m=+2566.279179840" watchObservedRunningTime="2026-11-03 16:57:50.959894877 +0000 UTC m=+2566.293108446" Nov 03 16:57:51.623060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:51.623017 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:51.629616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:51.629587 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:51.894250 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:51.894147 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:51.895336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:51.895315 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:57:52.252371 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:57:52.252333 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:57:55.265651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.265614 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:57:55.270651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.270633 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.289701 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.289667 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:57:55.296972 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.296944 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c1837e\"" Nov 03 16:57:55.339374 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.339328 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksc9w\" (UniqueName: \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.339723 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.339696 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.460469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.458606 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.460469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.458824 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksc9w\" (UniqueName: \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.490471 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.481049 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.523530 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.523434 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksc9w\" (UniqueName: \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\") pod \"payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.588377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.588331 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:55.923382 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:57:55.923347 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aea26a3_d848_45fa_b170_cc0d98d525db.slice/crio-0e838bcf784fb755d7074e5638b740d7036346f1701a9cd45df93fb2d9a57df1 WatchSource:0}: Error finding container 0e838bcf784fb755d7074e5638b740d7036346f1701a9cd45df93fb2d9a57df1: Status 404 returned error can't find the container with id 0e838bcf784fb755d7074e5638b740d7036346f1701a9cd45df93fb2d9a57df1 Nov 03 16:57:55.924651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:55.924621 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:57:56.315817 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.315780 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" podUID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" containerName="main" containerID="cri-o://4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed" gracePeriod=45 Nov 03 16:57:56.657090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.657064 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.798669 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\" (UniqueName: \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.798747 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.798873 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.798924 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.799027 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.804486 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.799134 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\") pod \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\" (UID: \"e87be5c3-4695-4b68-86a4-4fc2b6e152d8\") " Nov 03 16:57:56.814093 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.814030 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:56.814704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.814663 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:56.817073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.815536 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:56.819540 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.819510 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "kube-api-access-mhzf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:56.819970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.819942 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:57:56.820122 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.819999 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs" pod "e87be5c3-4695-4b68-86a4-4fc2b6e152d8" (UID: "e87be5c3-4695-4b68-86a4-4fc2b6e152d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915543 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzf4\" (UniqueName: \"kubernetes.io/projected/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-kube-api-access-mhzf4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915575 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915587 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915596 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915606 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.915637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.915614 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e87be5c3-4695-4b68-86a4-4fc2b6e152d8-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:57:56.939068 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.939032 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" event={"ID":"4aea26a3-d848-45fa-b170-cc0d98d525db","Type":"ContainerStarted","Data":"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a"} Nov 03 16:57:56.939255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.939090 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" event={"ID":"4aea26a3-d848-45fa-b170-cc0d98d525db","Type":"ContainerStarted","Data":"0e838bcf784fb755d7074e5638b740d7036346f1701a9cd45df93fb2d9a57df1"} Nov 03 16:57:56.939255 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.939118 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:57:56.940562 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.940534 6001 generic.go:356] "Generic (PLEG): container finished" podID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" containerID="4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed" exitCode=0 Nov 03 16:57:56.940690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.940575 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" Nov 03 16:57:56.940690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.940597 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" event={"ID":"e87be5c3-4695-4b68-86a4-4fc2b6e152d8","Type":"ContainerDied","Data":"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed"} Nov 03 16:57:56.940690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.940637 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw" event={"ID":"e87be5c3-4695-4b68-86a4-4fc2b6e152d8","Type":"ContainerDied","Data":"11832adba4445a4e8df488934acc92c118bade926a596439cb4bcadc164db495"} Nov 03 16:57:56.940690 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.940658 6001 scope.go:122] "RemoveContainer" containerID="4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed" Nov 03 16:57:56.952455 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.952421 6001 scope.go:122] "RemoveContainer" containerID="4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed" Nov 03 16:57:56.952798 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:57:56.952779 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed\": container with ID starting with 4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed not found: ID does not exist" containerID="4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed" Nov 03 16:57:56.952847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.952809 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed"} err="failed to get container status \"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed\": rpc error: code = NotFound desc = could not find container \"4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed\": container with ID starting with 4e3ed755eae8b288d464cf007cfc3567e3d5a5076853f87f3000583cdde321ed not found: ID does not exist" Nov 03 16:57:56.985125 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:56.985064 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" podStartSLOduration=1.9850453049999999 podStartE2EDuration="1.985045305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:56.98119343 +0000 UTC m=+2572.314406999" watchObservedRunningTime="2026-11-03 16:57:56.985045305 +0000 UTC m=+2572.318258881" Nov 03 16:57:57.024112 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:57.024051 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:57:57.028986 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:57.028964 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5a5ab1/auth-test-model-5a5ab1-kserve-78448f77-jz6rw"] Nov 03 16:57:58.791660 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:58.791626 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" path="/var/lib/kubelet/pods/e87be5c3-4695-4b68-86a4-4fc2b6e152d8/volumes" Nov 03 16:57:59.033327 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:59.033291 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5"] Nov 03 16:57:59.045898 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:59.045777 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8"] Nov 03 16:57:59.050057 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:59.050027 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fca59b-x5lr5"] Nov 03 16:57:59.054637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:57:59.054612 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b27195cc-lj2l8"] Nov 03 16:58:00.339053 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.339020 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:00.340097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.340073 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" containerName="main" Nov 03 16:58:00.340097 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.340098 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" containerName="main" Nov 03 16:58:00.340309 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.340286 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87be5c3-4695-4b68-86a4-4fc2b6e152d8" containerName="main" Nov 03 16:58:00.345276 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.345252 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.370527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.370492 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:00.376072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.376044 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-43b1d1-openshift-default-dockercfg-2dcvl\"" Nov 03 16:58:00.431296 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431256 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431510 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431484 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431578 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431526 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431578 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431556 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431727 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431703 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431878 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431857 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76qkf\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.431984 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.431968 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.432025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.432011 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.432069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.432038 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549479 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549652 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549694 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549723 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549775 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.549924 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76qkf\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.550022 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.550065 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.550096 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.551346 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.550947 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.573468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.552777 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.573468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.553435 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.573468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.554010 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.573468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.571003 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.574224 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.574189 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.609519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.605386 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.609519 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.608247 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.656903 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.656822 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76qkf\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\") pod \"e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.675121 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.674779 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:00.792475 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.792400 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44db7791-687e-43ae-95a5-21c828d190a4" path="/var/lib/kubelet/pods/44db7791-687e-43ae-95a5-21c828d190a4/volumes" Nov 03 16:58:00.793132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:00.793106 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7951fd-5622-4286-a054-47cd5e5e0bdb" path="/var/lib/kubelet/pods/7d7951fd-5622-4286-a054-47cd5e5e0bdb/volumes" Nov 03 16:58:01.069343 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:58:01.069311 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc944945f_ce11_4227_95a1_c587175a64ce.slice/crio-d60a304eec0683f0a63ce0b28fe628d07bf00b38dff8914c5ff3b6cc4dce48c0 WatchSource:0}: Error finding container d60a304eec0683f0a63ce0b28fe628d07bf00b38dff8914c5ff3b6cc4dce48c0: Status 404 returned error can't find the container with id d60a304eec0683f0a63ce0b28fe628d07bf00b38dff8914c5ff3b6cc4dce48c0 Nov 03 16:58:01.071157 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.071127 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:01.072473 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.072415 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:01.072547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.072503 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:01.072547 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.072535 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:01.987453 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.987413 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" event={"ID":"c944945f-ce11-4227-95a1-c587175a64ce","Type":"ContainerStarted","Data":"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93"} Nov 03 16:58:01.987928 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:01.987489 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" event={"ID":"c944945f-ce11-4227-95a1-c587175a64ce","Type":"ContainerStarted","Data":"d60a304eec0683f0a63ce0b28fe628d07bf00b38dff8914c5ff3b6cc4dce48c0"} Nov 03 16:58:02.055101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:02.055041 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" podStartSLOduration=2.055025231 podStartE2EDuration="2.055025231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:02.048465656 +0000 UTC m=+2577.381679226" watchObservedRunningTime="2026-11-03 16:58:02.055025231 +0000 UTC m=+2577.388238799" Nov 03 16:58:02.675289 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:02.675251 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:02.681946 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:02.681916 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:02.996042 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:02.996014 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:02.996989 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:02.996969 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:07.954194 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:07.954165 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:58:09.032184 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:09.032153 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz"] Nov 03 16:58:09.046851 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:09.046815 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff11751-bgwfz"] Nov 03 16:58:09.824620 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:09.824584 6001 scope.go:122] "RemoveContainer" containerID="c501a72823eab37a9530cca6539fd4f0a81a2026686530bc3802857529e0d4ce" Nov 03 16:58:09.836384 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:09.836361 6001 scope.go:122] "RemoveContainer" containerID="53bc2c93b3aa07c2a8e15ca371cfca4508982a0a1648f0fb4c5818a88cee1afd" Nov 03 16:58:09.846770 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:09.846752 6001 scope.go:122] "RemoveContainer" containerID="a9430da21e5580cbc7bf715aa906b04ca779b768648001e23ab1a16876857056" Nov 03 16:58:10.791018 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:10.790984 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1015cf7-c27c-4918-860f-90d4af34e916" path="/var/lib/kubelet/pods/f1015cf7-c27c-4918-860f-90d4af34e916/volumes" Nov 03 16:58:12.036341 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:12.036304 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4"] Nov 03 16:58:12.048969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:12.048939 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b76ef2b-kv7v4"] Nov 03 16:58:12.792070 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:12.792030 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2098b23-6f88-4ed0-97f2-f69170e4b8ac" path="/var/lib/kubelet/pods/c2098b23-6f88-4ed0-97f2-f69170e4b8ac/volumes" Nov 03 16:58:16.043354 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:16.043316 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd"] Nov 03 16:58:16.056377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:16.056339 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f9a7cb-vcsvd"] Nov 03 16:58:16.790487 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:16.790436 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df29677a-31e0-47ee-bbd5-923bad224384" path="/var/lib/kubelet/pods/df29677a-31e0-47ee-bbd5-923bad224384/volumes" Nov 03 16:58:22.253274 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:58:22.253235 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:58:25.036338 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:25.036298 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4"] Nov 03 16:58:25.046623 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:25.046583 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-068f5bc3-g22m4"] Nov 03 16:58:26.791279 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:26.791243 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c387f6d-864c-44c9-8ab4-a63e0c112bb2" path="/var/lib/kubelet/pods/8c387f6d-864c-44c9-8ab4-a63e0c112bb2/volumes" Nov 03 16:58:27.619208 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:27.619146 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:27.619632 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:27.619585 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" podUID="c944945f-ce11-4227-95a1-c587175a64ce" containerName="istio-proxy" containerID="cri-o://fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93" gracePeriod=30 Nov 03 16:58:29.045733 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:29.045694 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf"] Nov 03 16:58:29.058107 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:29.058081 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90950915-7qxwf"] Nov 03 16:58:30.790704 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:30.790669 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30" path="/var/lib/kubelet/pods/e72e0a0c-3f3f-4e5f-a068-8aa8ba9dca30/volumes" Nov 03 16:58:33.461236 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.461150 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:33.474668 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.474627 6001 generic.go:356] "Generic (PLEG): container finished" podID="c944945f-ce11-4227-95a1-c587175a64ce" containerID="fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93" exitCode=0 Nov 03 16:58:33.475127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.474715 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" event={"ID":"c944945f-ce11-4227-95a1-c587175a64ce","Type":"ContainerDied","Data":"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93"} Nov 03 16:58:33.475297 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.475280 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" event={"ID":"c944945f-ce11-4227-95a1-c587175a64ce","Type":"ContainerDied","Data":"d60a304eec0683f0a63ce0b28fe628d07bf00b38dff8914c5ff3b6cc4dce48c0"} Nov 03 16:58:33.475394 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.475347 6001 scope.go:122] "RemoveContainer" containerID="fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93" Nov 03 16:58:33.489474 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.489433 6001 scope.go:122] "RemoveContainer" containerID="fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93" Nov 03 16:58:33.489854 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:58:33.489827 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93\": container with ID starting with fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93 not found: ID does not exist" containerID="fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93" Nov 03 16:58:33.489973 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.489862 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93"} err="failed to get container status \"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93\": rpc error: code = NotFound desc = could not find container \"fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93\": container with ID starting with fc66fd7994081fb369cb3fe526e8f46aa5aa33d5dc766ffb52e557d5fc8ebd93 not found: ID does not exist" Nov 03 16:58:33.498651 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498618 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.498816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498669 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.498816 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498769 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.498940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498820 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.498940 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498929 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.499049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.498965 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.499049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499025 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.499153 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499061 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.499204 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499180 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\") pod \"c944945f-ce11-4227-95a1-c587175a64ce\" (UID: \"c944945f-ce11-4227-95a1-c587175a64ce\") " Nov 03 16:58:33.499357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499323 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:33.499545 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499517 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:33.499637 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499546 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:33.499711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499625 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:33.499711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499669 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:33.500095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.499984 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c944945f-ce11-4227-95a1-c587175a64ce-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.500095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.500011 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.500095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.500036 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.500095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.500051 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.500095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.500064 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.501764 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.501678 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "kube-api-access-76qkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:33.501873 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.501803 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:33.502142 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.502111 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:33.503066 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.503044 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo" pod "c944945f-ce11-4227-95a1-c587175a64ce" (UID: "c944945f-ce11-4227-95a1-c587175a64ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:58:33.601997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.601953 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76qkf\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-kube-api-access-76qkf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.601997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.601988 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c944945f-ce11-4227-95a1-c587175a64ce-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.601997 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.602001 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c944945f-ce11-4227-95a1-c587175a64ce-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:33.602252 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:33.602013 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c944945f-ce11-4227-95a1-c587175a64ce-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:58:34.181170 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.181112 6001 patch_prober.go:29] interesting pod/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:58:34.181370 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.181197 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" podUID="c944945f-ce11-4227-95a1-c587175a64ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.238:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:58:34.484931 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.484896 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk" Nov 03 16:58:34.557034 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.556996 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:34.571209 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.571171 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-43b1d1-openshift-default-6b674f9487-gfrhk"] Nov 03 16:58:34.791927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:34.791844 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c944945f-ce11-4227-95a1-c587175a64ce" path="/var/lib/kubelet/pods/c944945f-ce11-4227-95a1-c587175a64ce/volumes" Nov 03 16:58:40.031509 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:40.031467 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9"] Nov 03 16:58:40.043768 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:40.043734 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887accd4-2t5p9"] Nov 03 16:58:40.792323 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:40.792284 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403af292-5eea-4b4b-bd5a-5b496f5af605" path="/var/lib/kubelet/pods/403af292-5eea-4b4b-bd5a-5b496f5af605/volumes" Nov 03 16:58:44.073923 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:44.073880 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9"] Nov 03 16:58:44.077840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:44.077815 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb42f7c0-76qm9"] Nov 03 16:58:44.794123 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:44.794090 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f7a226-f383-49e7-9c5f-dfa0eea71933" path="/var/lib/kubelet/pods/57f7a226-f383-49e7-9c5f-dfa0eea71933/volumes" Nov 03 16:58:50.032358 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:50.032313 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5"] Nov 03 16:58:50.044953 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:50.044909 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9166371b-95mp5"] Nov 03 16:58:50.791431 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:50.791394 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acaf0059-e09c-4b10-9a00-4971b7480aa0" path="/var/lib/kubelet/pods/acaf0059-e09c-4b10-9a00-4971b7480aa0/volumes" Nov 03 16:58:52.252729 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:58:52.252687 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:58:52.935619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:52.935579 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:58:52.935847 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:52.935672 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:58:54.602630 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.602582 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 16:58:54.603411 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.603392 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c944945f-ce11-4227-95a1-c587175a64ce" containerName="istio-proxy" Nov 03 16:58:54.603499 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.603412 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c944945f-ce11-4227-95a1-c587175a64ce" containerName="istio-proxy" Nov 03 16:58:54.603582 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.603571 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c944945f-ce11-4227-95a1-c587175a64ce" containerName="istio-proxy" Nov 03 16:58:54.669525 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.669491 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 16:58:54.669736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.669630 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.686277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.686156 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3b6f35-openshift-default-dockercfg-jr4pt\"" Nov 03 16:58:54.719114 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719064 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719134 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719342 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719600 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719573 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719753 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719843 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719820 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.719945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.720011 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.719978 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsj5b\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.720082 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.720062 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.821960 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822092 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822213 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822191 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822232 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsj5b\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822274 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822310 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822407 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822342 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822406 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822412 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822763 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822749 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822807 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.822945 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.822881 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.823073 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.823041 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.824859 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.824828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.824969 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.824935 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.856267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.850315 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:54.856267 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:54.850419 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsj5b\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\") pod \"e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:55.011071 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.011035 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:55.342418 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:58:55.342382 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cc4377_5e92_4fcb_abb0_e79d61146f86.slice/crio-0cc78963b08fdfcdee4d0dd43394ed8fc20d6cae424a671fd01e2814facc342a WatchSource:0}: Error finding container 0cc78963b08fdfcdee4d0dd43394ed8fc20d6cae424a671fd01e2814facc342a: Status 404 returned error can't find the container with id 0cc78963b08fdfcdee4d0dd43394ed8fc20d6cae424a671fd01e2814facc342a Nov 03 16:58:55.342840 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.342816 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 16:58:55.345391 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.345350 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.345527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.345427 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.345527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.345478 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.758025 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.757988 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" event={"ID":"63cc4377-5e92-4fcb-abb0-e79d61146f86","Type":"ContainerStarted","Data":"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d"} Nov 03 16:58:55.758531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.758051 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" event={"ID":"63cc4377-5e92-4fcb-abb0-e79d61146f86","Type":"ContainerStarted","Data":"0cc78963b08fdfcdee4d0dd43394ed8fc20d6cae424a671fd01e2814facc342a"} Nov 03 16:58:55.760090 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.760062 6001 generic.go:356] "Generic (PLEG): container finished" podID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerID="aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9" exitCode=0 Nov 03 16:58:55.760283 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.760135 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerDied","Data":"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9"} Nov 03 16:58:55.760611 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.760595 6001 scope.go:122] "RemoveContainer" containerID="aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9" Nov 03 16:58:55.762414 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.762381 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.762539 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.762483 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.762621 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.762537 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:58:55.829469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:55.829391 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podStartSLOduration=1.829375009 podStartE2EDuration="1.829375009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:55.824750369 +0000 UTC m=+2631.157963973" watchObservedRunningTime="2026-11-03 16:58:55.829375009 +0000 UTC m=+2631.162588578" Nov 03 16:58:56.012317 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:56.012221 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:58:56.013824 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:56.013799 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:58:56.013970 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:56.013848 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:58:56.777871 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:56.777830 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerStarted","Data":"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c"} Nov 03 16:58:57.012280 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:57.012236 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:58:57.012556 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:57.012325 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:58:57.623866 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:57.623823 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:58:57.624069 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:57.624044 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:58:57.624126 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:57.624104 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:58:58.011526 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:58.011488 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:58:58.011949 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:58.011550 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:58:58.623585 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:58.623539 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:58:58.623762 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:58.623605 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:58:59.012072 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.012034 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:58:59.012536 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.012089 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:58:59.030751 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.030716 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn"] Nov 03 16:58:59.047470 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.044053 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20fac21b-5zbtn"] Nov 03 16:58:59.623882 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.623845 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:58:59.624084 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:58:59.623903 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:59:00.020555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:00.020240 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:59:00.020555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:00.020312 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:59:00.623313 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:00.623283 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:59:00.623819 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:00.623789 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:59:00.814458 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:00.814411 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edc192b-af7f-495e-8092-ca9facc4cbfc" path="/var/lib/kubelet/pods/1edc192b-af7f-495e-8092-ca9facc4cbfc/volumes" Nov 03 16:59:01.011531 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:01.011494 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:59:01.011714 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:01.011561 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:59:01.624329 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:01.624289 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:59:01.624869 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:01.624370 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:59:02.012393 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:02.012348 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Nov 03 16:59:02.012616 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:02.012435 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Nov 03 16:59:02.623482 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:02.623430 6001 patch_prober.go:29] interesting pod/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" start-of-body= Nov 03 16:59:02.623670 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:02.623518 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.236:15021/healthz/ready\": dial tcp 10.134.0.236:15021: connect: connection refused" Nov 03 16:59:03.018145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.018111 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:59:03.018724 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.018660 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:59:03.019340 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.019323 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 16:59:03.033872 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.033845 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm"] Nov 03 16:59:03.039409 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.039380 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7d2f268-qzhbm"] Nov 03 16:59:03.628648 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.628615 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:59:03.629146 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.629119 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:59:03.630001 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:03.629983 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 16:59:04.986468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:04.970614 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f47deb0b-4a5b-4feb-8a32-6c9325fe43d4" path="/var/lib/kubelet/pods/f47deb0b-4a5b-4feb-8a32-6c9325fe43d4/volumes" Nov 03 16:59:09.033988 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.033951 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s"] Nov 03 16:59:09.046517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.046485 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655bbfa7-kxs9s"] Nov 03 16:59:09.460377 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.460270 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58"] Nov 03 16:59:09.466864 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.466835 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.488242 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.488209 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58"] Nov 03 16:59:09.497076 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.497045 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:09.497257 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.497156 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:09.501129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.501108 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:09.501129 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.501133 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:59:09.612427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.612389 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndmdg\" (UniqueName: \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.612427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.612466 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.612682 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.612644 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.714060 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.713968 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.714241 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.714223 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndmdg\" (UniqueName: \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.714315 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.714297 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.714814 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.714791 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.716832 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.716808 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.739779 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.739753 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndmdg\" (UniqueName: \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\") pod \"maas-api-revoke-sub-38680f90-lmz58\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.781420 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.781395 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:09.904821 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.904782 6001 scope.go:122] "RemoveContainer" containerID="4ad0a41813b9b83de450b27993221bf3b09f1bdfc100d87a605cc3e7fcd086fe" Nov 03 16:59:09.918012 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.917995 6001 scope.go:122] "RemoveContainer" containerID="3e1059ccf8a3822f6486fcfab26a966a04a331f034060d845b195f49c12e7fa4" Nov 03 16:59:09.931870 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.931141 6001 scope.go:122] "RemoveContainer" containerID="32b977b1b355ef57adcbd9ff5a4a4dd3e48e8f6fe7e6933eb78a5c0ff030997a" Nov 03 16:59:09.944461 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.944427 6001 scope.go:122] "RemoveContainer" containerID="ad6a17fd39bd705811ad05a9a067aea617483c05c644291a011d90d83eea179a" Nov 03 16:59:09.954849 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.954830 6001 scope.go:122] "RemoveContainer" containerID="934b48f6702d38e807d942e0366f58954e045559fe27bbc9caf452024390db19" Nov 03 16:59:09.968935 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.968911 6001 scope.go:122] "RemoveContainer" containerID="6dda6b1c99c67dea738d290fe9c2a727e1d7bdda17afdc2baf4103ddce2904b4" Nov 03 16:59:09.980619 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:09.980597 6001 scope.go:122] "RemoveContainer" containerID="dbd85626c3de6da98ce243023e6a753ee7489b34717d2c7f7d90e4ed6a92b666" Nov 03 16:59:10.094563 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:59:10.094529 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a548ba_0060_4691_82b0_b9fe4e050182.slice/crio-d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678 WatchSource:0}: Error finding container d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678: Status 404 returned error can't find the container with id d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678 Nov 03 16:59:10.095183 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:10.095157 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58"] Nov 03 16:59:10.792683 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:10.792653 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40d3cc9-878a-4784-822b-a2eda88ff49d" path="/var/lib/kubelet/pods/f40d3cc9-878a-4784-822b-a2eda88ff49d/volumes" Nov 03 16:59:11.055324 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:11.055292 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" event={"ID":"f6a548ba-0060-4691-82b0-b9fe4e050182","Type":"ContainerStarted","Data":"e734d8ee9a663874ec2bfe9951f9607e5e896701a4bbde157b783116c9edca7e"} Nov 03 16:59:11.055497 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:11.055350 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" event={"ID":"f6a548ba-0060-4691-82b0-b9fe4e050182","Type":"ContainerStarted","Data":"d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678"} Nov 03 16:59:11.061872 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:59:11.061844 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a548ba_0060_4691_82b0_b9fe4e050182.slice/crio-e734d8ee9a663874ec2bfe9951f9607e5e896701a4bbde157b783116c9edca7e.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:59:11.102273 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:11.102227 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" podStartSLOduration=2.10221433 podStartE2EDuration="2.10221433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:11.098209049 +0000 UTC m=+2646.431422619" watchObservedRunningTime="2026-11-03 16:59:11.10221433 +0000 UTC m=+2646.435427898" Nov 03 16:59:12.065101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:12.065063 6001 generic.go:356] "Generic (PLEG): container finished" podID="f6a548ba-0060-4691-82b0-b9fe4e050182" containerID="e734d8ee9a663874ec2bfe9951f9607e5e896701a4bbde157b783116c9edca7e" exitCode=0 Nov 03 16:59:12.065101 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:12.065096 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" event={"ID":"f6a548ba-0060-4691-82b0-b9fe4e050182","Type":"ContainerDied","Data":"e734d8ee9a663874ec2bfe9951f9607e5e896701a4bbde157b783116c9edca7e"} Nov 03 16:59:13.247357 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.247323 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:13.371886 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.371843 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\" (UniqueName: \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\") pod \"f6a548ba-0060-4691-82b0-b9fe4e050182\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " Nov 03 16:59:13.372127 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.372108 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\") pod \"f6a548ba-0060-4691-82b0-b9fe4e050182\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " Nov 03 16:59:13.372191 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.372152 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\") pod \"f6a548ba-0060-4691-82b0-b9fe4e050182\" (UID: \"f6a548ba-0060-4691-82b0-b9fe4e050182\") " Nov 03 16:59:13.372555 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.372505 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca" pod "f6a548ba-0060-4691-82b0-b9fe4e050182" (UID: "f6a548ba-0060-4691-82b0-b9fe4e050182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:13.372893 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.372872 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:13.374217 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.374193 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg" pod "f6a548ba-0060-4691-82b0-b9fe4e050182" (UID: "f6a548ba-0060-4691-82b0-b9fe4e050182"). InnerVolumeSpecName "kube-api-access-ndmdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:13.374314 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.374290 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca" pod "f6a548ba-0060-4691-82b0-b9fe4e050182" (UID: "f6a548ba-0060-4691-82b0-b9fe4e050182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:13.474681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.474641 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndmdg\" (UniqueName: \"kubernetes.io/projected/f6a548ba-0060-4691-82b0-b9fe4e050182-kube-api-access-ndmdg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:13.474681 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:13.474680 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a548ba-0060-4691-82b0-b9fe4e050182-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:14.082711 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:14.082682 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" Nov 03 16:59:14.082942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:14.082705 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58" event={"ID":"f6a548ba-0060-4691-82b0-b9fe4e050182","Type":"ContainerDied","Data":"d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678"} Nov 03 16:59:14.082942 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:14.082764 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d14414ac056987261a19687d2ed13cd74acef3bd69b18ba91e47ec0cd3389678" Nov 03 16:59:15.035998 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.035959 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d"] Nov 03 16:59:15.048277 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.048243 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54eaf8bd-svg8d"] Nov 03 16:59:15.505941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.505903 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 16:59:15.507160 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.507135 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f6a548ba-0060-4691-82b0-b9fe4e050182" containerName="revoke-keys" Nov 03 16:59:15.507214 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.507164 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a548ba-0060-4691-82b0-b9fe4e050182" containerName="revoke-keys" Nov 03 16:59:15.507472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.507456 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a548ba-0060-4691-82b0-b9fe4e050182" containerName="revoke-keys" Nov 03 16:59:15.511472 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.511435 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.530454 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.530409 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a9426b-openshift-default-dockercfg-tv556\"" Nov 03 16:59:15.552137 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.552095 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 16:59:15.608736 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.608698 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcr4t\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.608927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.608741 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.608927 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.608858 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609032 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609012 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609120 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609107 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609155 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609139 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609193 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609185 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609253 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609240 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.609291 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.609272 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.719614 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.719834 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.719948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.719990 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.720060 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.720099 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.720139 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.720245 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcr4t\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.720278 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.721483 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.721743 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.723469 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.723317 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.724698 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.723611 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.745580 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.732202 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.755484 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.751288 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.787517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.783697 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.787517 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.783997 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.821467 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.817340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcr4t\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:15.836897 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:15.830314 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:16.040941 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.040849 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z"] Nov 03 16:59:16.045722 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.045689 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c1bc435-s5x5z"] Nov 03 16:59:16.232979 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:59:16.232946 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3844c53f_0103_4055_8a03_9416753e16e4.slice/crio-6a8e5feeb23e914978a6cf686b33434f73404d8cd6dfb21cb3745606971ec916 WatchSource:0}: Error finding container 6a8e5feeb23e914978a6cf686b33434f73404d8cd6dfb21cb3745606971ec916: Status 404 returned error can't find the container with id 6a8e5feeb23e914978a6cf686b33434f73404d8cd6dfb21cb3745606971ec916 Nov 03 16:59:16.233695 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.233669 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 16:59:16.235729 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.235693 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:59:16.235848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.235773 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:59:16.235848 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.235818 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 16:59:16.805239 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.805200 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b7d753-c42b-4903-bbe9-daa084e1f230" path="/var/lib/kubelet/pods/78b7d753-c42b-4903-bbe9-daa084e1f230/volumes" Nov 03 16:59:16.805772 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:16.805757 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daaa5361-c760-4b89-ab2c-7e43e01967ff" path="/var/lib/kubelet/pods/daaa5361-c760-4b89-ab2c-7e43e01967ff/volumes" Nov 03 16:59:17.109332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:17.109233 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerStarted","Data":"0e354216adbad57e9e1302256588b77aeb4ebb787a327b6e384e3706d472ae56"} Nov 03 16:59:17.109332 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:17.109305 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerStarted","Data":"6a8e5feeb23e914978a6cf686b33434f73404d8cd6dfb21cb3745606971ec916"} Nov 03 16:59:17.191679 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:17.191622 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podStartSLOduration=2.19160633 podStartE2EDuration="2.19160633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:17.175759996 +0000 UTC m=+2652.508973578" watchObservedRunningTime="2026-11-03 16:59:17.19160633 +0000 UTC m=+2652.524819973" Nov 03 16:59:17.847468 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:17.841825 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:17.951139 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:17.948543 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:18.118145 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:18.118068 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:18.119087 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:18.119068 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 16:59:19.843836 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.843802 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x"] Nov 03 16:59:19.849844 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.849825 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:19.868566 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.868538 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 16:59:19.868731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.868590 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:59:19.868731 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.868697 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:59:19.870907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.870889 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x"] Nov 03 16:59:19.878765 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.878746 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 16:59:19.897605 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.897572 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:19.898132 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.898101 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbmtx\" (UniqueName: \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:19.898318 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:19.898267 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.000095 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.000065 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.000287 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.000223 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbmtx\" (UniqueName: \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.000356 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.000286 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.000954 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.000935 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.002561 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.002531 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.028105 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.028074 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbmtx\" (UniqueName: \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\") pod \"maas-api-revoke-sub-9b024d73-lfd6x\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.165702 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.165623 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:20.487161 ip-10-0-129-130 kubenswrapper[6001]: W1103 16:59:20.487129 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5a3d56_d0c0_4199_b5c7_44439436b279.slice/crio-51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332 WatchSource:0}: Error finding container 51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332: Status 404 returned error can't find the container with id 51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332 Nov 03 16:59:20.487907 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:20.487882 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x"] Nov 03 16:59:21.150656 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:21.150498 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" event={"ID":"1e5a3d56-d0c0-4199-b5c7-44439436b279","Type":"ContainerStarted","Data":"283af6d8fb7cc764030243f7c929680122607bd380db079e577f78f20e7adc6f"} Nov 03 16:59:21.151210 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:21.151179 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" event={"ID":"1e5a3d56-d0c0-4199-b5c7-44439436b279","Type":"ContainerStarted","Data":"51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332"} Nov 03 16:59:21.205936 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:21.205862 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" podStartSLOduration=2.20584439 podStartE2EDuration="2.20584439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:21.202684223 +0000 UTC m=+2656.535897804" watchObservedRunningTime="2026-11-03 16:59:21.20584439 +0000 UTC m=+2656.539057963" Nov 03 16:59:22.163262 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:22.163226 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e5a3d56-d0c0-4199-b5c7-44439436b279" containerID="283af6d8fb7cc764030243f7c929680122607bd380db079e577f78f20e7adc6f" exitCode=0 Nov 03 16:59:22.163757 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:22.163312 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" event={"ID":"1e5a3d56-d0c0-4199-b5c7-44439436b279","Type":"ContainerDied","Data":"283af6d8fb7cc764030243f7c929680122607bd380db079e577f78f20e7adc6f"} Nov 03 16:59:22.259419 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:59:22.259380 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:59:23.354040 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.354010 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:23.414223 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.414166 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\") pod \"1e5a3d56-d0c0-4199-b5c7-44439436b279\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " Nov 03 16:59:23.414793 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.414761 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\") pod \"1e5a3d56-d0c0-4199-b5c7-44439436b279\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " Nov 03 16:59:23.414991 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.414974 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\" (UniqueName: \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\") pod \"1e5a3d56-d0c0-4199-b5c7-44439436b279\" (UID: \"1e5a3d56-d0c0-4199-b5c7-44439436b279\") " Nov 03 16:59:23.431049 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.430613 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx" pod "1e5a3d56-d0c0-4199-b5c7-44439436b279" (UID: "1e5a3d56-d0c0-4199-b5c7-44439436b279"). InnerVolumeSpecName "kube-api-access-bbmtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:23.435828 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.435775 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca" pod "1e5a3d56-d0c0-4199-b5c7-44439436b279" (UID: "1e5a3d56-d0c0-4199-b5c7-44439436b279"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:23.440987 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.440929 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca" pod "1e5a3d56-d0c0-4199-b5c7-44439436b279" (UID: "1e5a3d56-d0c0-4199-b5c7-44439436b279"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:23.525527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.525492 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:23.525527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.525525 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5a3d56-d0c0-4199-b5c7-44439436b279-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:23.525527 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:23.525536 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbmtx\" (UniqueName: \"kubernetes.io/projected/1e5a3d56-d0c0-4199-b5c7-44439436b279-kube-api-access-bbmtx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.181839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:24.181807 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" Nov 03 16:59:24.181839 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:24.181832 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x" event={"ID":"1e5a3d56-d0c0-4199-b5c7-44439436b279","Type":"ContainerDied","Data":"51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332"} Nov 03 16:59:24.182094 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:24.181879 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51b5390693e34ab1fdeddbdce43d626f372113701871838cbe353dd03abf4332" Nov 03 16:59:40.035085 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:40.035048 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:59:45.036978 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.036911 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" podUID="4aea26a3-d848-45fa-b170-cc0d98d525db" containerName="payload-processing" containerID="cri-o://1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a" gracePeriod=25 Nov 03 16:59:45.346981 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.346953 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:59:45.434365 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.434319 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\") pod \"4aea26a3-d848-45fa-b170-cc0d98d525db\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " Nov 03 16:59:45.434846 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.434816 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume" pod "4aea26a3-d848-45fa-b170-cc0d98d525db" (UID: "4aea26a3-d848-45fa-b170-cc0d98d525db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:45.435038 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.434909 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\" (UniqueName: \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\") pod \"4aea26a3-d848-45fa-b170-cc0d98d525db\" (UID: \"4aea26a3-d848-45fa-b170-cc0d98d525db\") " Nov 03 16:59:45.436023 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.435981 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aea26a3-d848-45fa-b170-cc0d98d525db-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:45.445117 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.445056 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w" pod "4aea26a3-d848-45fa-b170-cc0d98d525db" (UID: "4aea26a3-d848-45fa-b170-cc0d98d525db"). InnerVolumeSpecName "kube-api-access-ksc9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:45.549546 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.549481 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksc9w\" (UniqueName: \"kubernetes.io/projected/4aea26a3-d848-45fa-b170-cc0d98d525db-kube-api-access-ksc9w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:45.566298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.566207 6001 generic.go:356] "Generic (PLEG): container finished" podID="4aea26a3-d848-45fa-b170-cc0d98d525db" containerID="1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a" exitCode=0 Nov 03 16:59:45.566298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.566265 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" Nov 03 16:59:45.566298 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.566286 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" event={"ID":"4aea26a3-d848-45fa-b170-cc0d98d525db","Type":"ContainerDied","Data":"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a"} Nov 03 16:59:45.566589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.566350 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln" event={"ID":"4aea26a3-d848-45fa-b170-cc0d98d525db","Type":"ContainerDied","Data":"0e838bcf784fb755d7074e5638b740d7036346f1701a9cd45df93fb2d9a57df1"} Nov 03 16:59:45.566589 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.566380 6001 scope.go:122] "RemoveContainer" containerID="1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a" Nov 03 16:59:45.580249 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.580223 6001 scope.go:122] "RemoveContainer" containerID="1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a" Nov 03 16:59:45.580707 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:59:45.580680 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a\": container with ID starting with 1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a not found: ID does not exist" containerID="1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a" Nov 03 16:59:45.580796 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.580719 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a"} err="failed to get container status \"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a\": rpc error: code = NotFound desc = could not find container \"1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a\": container with ID starting with 1e16a776d79713df5f7ad5de99de971db4092dd9ed8f9ed3cb2ac854945a9b5a not found: ID does not exist" Nov 03 16:59:45.623995 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.623953 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:59:45.628231 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:45.628200 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-c1837e-6b4fd9d84c-n8vln"] Nov 03 16:59:46.791532 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:46.791497 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aea26a3-d848-45fa-b170-cc0d98d525db" path="/var/lib/kubelet/pods/4aea26a3-d848-45fa-b170-cc0d98d525db/volumes" Nov 03 16:59:49.453780 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:49.453735 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:59:49.454251 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:49.453977 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" containerID="cri-o://24ca42c1e3a2b597730f56af84e94bdc834e3e67e640b7fba3769517ea83d8de" gracePeriod=30 Nov 03 16:59:52.253409 ip-10-0-129-130 kubenswrapper[6001]: E1103 16:59:52.253369 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 16:59:54.694003 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.693967 6001 generic.go:356] "Generic (PLEG): container finished" podID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerID="24ca42c1e3a2b597730f56af84e94bdc834e3e67e640b7fba3769517ea83d8de" exitCode=0 Nov 03 16:59:54.694542 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.694046 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" event={"ID":"85a52f3a-2bd0-4bdf-9181-9719d241a0f7","Type":"ContainerDied","Data":"24ca42c1e3a2b597730f56af84e94bdc834e3e67e640b7fba3769517ea83d8de"} Nov 03 16:59:54.798359 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.798331 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:59:54.975842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.975808 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.975842 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.975848 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.975968 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976030 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976106 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976049 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976109 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976174 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:54.976336 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976300 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:54.976568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976317 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:54.976568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976405 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:54.976568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976221 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976568 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976543 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\") pod \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\" (UID: \"85a52f3a-2bd0-4bdf-9181-9719d241a0f7\") " Nov 03 16:59:54.976852 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.976602 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:54.977428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.977321 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:54.977428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.977351 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:54.977428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.977378 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:54.977428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.977392 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:54.977428 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.977405 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:54.987143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.979870 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:54.987143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.983863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "kube-api-access-vr955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:54.987143 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.984783 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:54.989219 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:54.989186 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy" pod "85a52f3a-2bd0-4bdf-9181-9719d241a0f7" (UID: "85a52f3a-2bd0-4bdf-9181-9719d241a0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:55.079268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.079185 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr955\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-kube-api-access-vr955\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:55.079268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.079216 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:55.079268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.079228 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:55.079268 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.079236 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85a52f3a-2bd0-4bdf-9181-9719d241a0f7-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 16:59:55.703947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.703915 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" Nov 03 16:59:55.703947 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.703938 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4" event={"ID":"85a52f3a-2bd0-4bdf-9181-9719d241a0f7","Type":"ContainerDied","Data":"1b7d57e8c8f4e2221798d0e6577b3daa92e1b9d608e494b4e2e1575e2b849822"} Nov 03 16:59:55.704427 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.704002 6001 scope.go:122] "RemoveContainer" containerID="24ca42c1e3a2b597730f56af84e94bdc834e3e67e640b7fba3769517ea83d8de" Nov 03 16:59:55.802827 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.802789 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:59:55.827115 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:55.827082 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-6lwq4"] Nov 03 16:59:56.791776 ip-10-0-129-130 kubenswrapper[6001]: I1103 16:59:56.791736 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" path="/var/lib/kubelet/pods/85a52f3a-2bd0-4bdf-9181-9719d241a0f7/volumes" Nov 03 17:00:00.232233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.232196 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:00.233376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233357 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4aea26a3-d848-45fa-b170-cc0d98d525db" containerName="payload-processing" Nov 03 17:00:00.233439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233379 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aea26a3-d848-45fa-b170-cc0d98d525db" containerName="payload-processing" Nov 03 17:00:00.233439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233411 6001 cpu_manager.go:414] "containerMap: removing container" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" Nov 03 17:00:00.233439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233420 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" Nov 03 17:00:00.233583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233470 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e5a3d56-d0c0-4199-b5c7-44439436b279" containerName="revoke-keys" Nov 03 17:00:00.233583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233480 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5a3d56-d0c0-4199-b5c7-44439436b279" containerName="revoke-keys" Nov 03 17:00:00.233653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233636 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aea26a3-d848-45fa-b170-cc0d98d525db" containerName="payload-processing" Nov 03 17:00:00.233690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233654 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a52f3a-2bd0-4bdf-9181-9719d241a0f7" containerName="istio-proxy" Nov 03 17:00:00.233690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.233667 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5a3d56-d0c0-4199-b5c7-44439436b279" containerName="revoke-keys" Nov 03 17:00:00.239975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.239902 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.245795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.245769 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:00.251347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.251316 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.252594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.252570 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:00.257543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.257514 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.258271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.258244 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdmw\" (UniqueName: \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.258528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.258386 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.261998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.261974 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:00.273063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.272994 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:00.278819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.278489 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:00:00.278819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.278638 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:00:00.278819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.278726 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:00:00.296208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.296168 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:00.296402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.296346 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.307169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.307082 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:00.307803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.307144 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:00.308027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.308009 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.315347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.315317 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:00.315908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.315463 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.321344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.321245 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:00.330123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.327545 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:00.347504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.346797 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:00.347504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.346954 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.365435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.365283 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:00.383594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.383018 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdmw\" (UniqueName: \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.383594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.383250 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.383594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.383429 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.383594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.383556 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp59z\" (UniqueName: \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.383966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.383669 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9v2\" (UniqueName: \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.384076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.384029 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.387844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.387810 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.390316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.390283 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:00.390412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.390402 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.393690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.393660 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:00.397679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.397650 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:00.403865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.403837 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:00.404065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.403992 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.411509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.411433 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:00.416645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.416622 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.423110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.423074 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:00.430122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.430088 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdmw\" (UniqueName: \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.469995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.469950 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:00.485425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485322 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxz7\" (UniqueName: \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.485425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485394 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9v2\" (UniqueName: \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.485686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485539 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.485686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485595 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.485800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485765 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7gq\" (UniqueName: \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.485971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.485942 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.486056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486038 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.486185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486160 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.486273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486250 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.486528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486401 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp59z\" (UniqueName: \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.486528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486495 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpg56\" (UniqueName: \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.487336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486846 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.487336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.486858 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb6lv\" (UniqueName: \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.487336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.487132 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.512930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.512895 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9v2\" (UniqueName: \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.518654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.518619 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp59z\" (UniqueName: \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.560501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.560464 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:00.588791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.588744 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lgxw\" (UniqueName: \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.588937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.588886 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.588982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.588964 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcdj9\" (UniqueName: \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.589043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589009 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpg56\" (UniqueName: \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.589141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589114 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb6lv\" (UniqueName: \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.589218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589202 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxz7\" (UniqueName: \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.589279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589246 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.589340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589327 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.589402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589372 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.589471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589406 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smr5b\" (UniqueName: \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.589615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589548 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7gq\" (UniqueName: \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.589719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589670 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.589860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589824 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.590025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.589889 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.590140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.590122 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.590665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.590609 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.590840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.590794 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.591087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.591052 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.594226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.594204 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:00.615653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.615619 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7gq\" (UniqueName: \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.615825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.615803 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxz7\" (UniqueName: \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.616241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.616214 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb6lv\" (UniqueName: \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.617568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.617541 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpg56\" (UniqueName: \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.650168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.649909 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:00.681483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.680773 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:00.701499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.695143 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smr5b\" (UniqueName: \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.701499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.695282 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.701499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.695398 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.701499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.695460 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lgxw\" (UniqueName: \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.701499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.695555 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcdj9\" (UniqueName: \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.702539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.702014 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.702539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.702301 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:00.703203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.703100 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:00.703203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.703158 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.703582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.703558 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.703673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.703581 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.745434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.745320 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcdj9\" (UniqueName: \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:00.745900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.745873 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lgxw\" (UniqueName: \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:00.764406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.757742 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smr5b\" (UniqueName: \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\") pod \"maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.786839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.779059 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:00.889005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.888733 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:00.910506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:00.904422 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:00.918730 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:00.918693 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e08b8db_cd60_4eb7_a590_3f3f0c6104ba.slice/crio-3f54ae4e7a520f87388d02203fffb7f98f05b16660e978c9b1f06321eba851c8 WatchSource:0}: Error finding container 3f54ae4e7a520f87388d02203fffb7f98f05b16660e978c9b1f06321eba851c8: Status 404 returned error can't find the container with id 3f54ae4e7a520f87388d02203fffb7f98f05b16660e978c9b1f06321eba851c8 Nov 03 17:00:01.016500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.015398 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:01.037017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.035573 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:01.056878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.053365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:01.071779 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.066703 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a09e64e5c453778d19e00341f3a0748b6a7ffad23750b4510480e5e597c7c3b9 WatchSource:0}: Error finding container a09e64e5c453778d19e00341f3a0748b6a7ffad23750b4510480e5e597c7c3b9: Status 404 returned error can't find the container with id a09e64e5c453778d19e00341f3a0748b6a7ffad23750b4510480e5e597c7c3b9 Nov 03 17:00:01.188838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.188746 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:01.190986 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.190946 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c51ae5a_63de_45e4_ad09_fecfba01f6fa.slice/crio-bb1748694fbc2d65106523cfa96e4f1e98bee71d5cb85f5bb30c9dda6ed6fb65 WatchSource:0}: Error finding container bb1748694fbc2d65106523cfa96e4f1e98bee71d5cb85f5bb30c9dda6ed6fb65: Status 404 returned error can't find the container with id bb1748694fbc2d65106523cfa96e4f1e98bee71d5cb85f5bb30c9dda6ed6fb65 Nov 03 17:00:01.329888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.329798 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:01.334971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.334944 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:01.338769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.338738 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:01.346676 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.346633 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d017ee1_1f5e_4ebc_814d_72db07b0f010.slice/crio-c5258f6829f5c7d43aaeb064cc715021365183a9ee3db29ca8b6d61b15fb9399 WatchSource:0}: Error finding container c5258f6829f5c7d43aaeb064cc715021365183a9ee3db29ca8b6d61b15fb9399: Status 404 returned error can't find the container with id c5258f6829f5c7d43aaeb064cc715021365183a9ee3db29ca8b6d61b15fb9399 Nov 03 17:00:01.348338 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.348293 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb50354_7d02_488d_b2d5_f44cd56f235c.slice/crio-995df06004f553150245a3670ded19812658f0759018a4b13bfda5914fe363b9 WatchSource:0}: Error finding container 995df06004f553150245a3670ded19812658f0759018a4b13bfda5914fe363b9: Status 404 returned error can't find the container with id 995df06004f553150245a3670ded19812658f0759018a4b13bfda5914fe363b9 Nov 03 17:00:01.350170 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.350137 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3d01a4_8458_4bbf_a98e_735627bfad04.slice/crio-61240c16b1fbf0497a5dcd66182be14cc3d35ca74f4b8ba9546f8be01d6c9b59 WatchSource:0}: Error finding container 61240c16b1fbf0497a5dcd66182be14cc3d35ca74f4b8ba9546f8be01d6c9b59: Status 404 returned error can't find the container with id 61240c16b1fbf0497a5dcd66182be14cc3d35ca74f4b8ba9546f8be01d6c9b59 Nov 03 17:00:01.374850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.374804 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:01.402672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.402642 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:01.427410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.427381 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:01.431645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.431618 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:01.452668 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.452622 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fc9ebc_6f86_4853_84ab_b09f38817d5c.slice/crio-7be8aab2f301932789fbd693fbf5d8b7b4856253372e98095f3097daeef4018c WatchSource:0}: Error finding container 7be8aab2f301932789fbd693fbf5d8b7b4856253372e98095f3097daeef4018c: Status 404 returned error can't find the container with id 7be8aab2f301932789fbd693fbf5d8b7b4856253372e98095f3097daeef4018c Nov 03 17:00:01.453586 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.453558 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26130612_35cb_44da_894e_9d243011d96d.slice/crio-681460a643991366b751ca5129f9afe37481be3b5b5f9e87c28a2cde148f4a06 WatchSource:0}: Error finding container 681460a643991366b751ca5129f9afe37481be3b5b5f9e87c28a2cde148f4a06: Status 404 returned error can't find the container with id 681460a643991366b751ca5129f9afe37481be3b5b5f9e87c28a2cde148f4a06 Nov 03 17:00:01.455005 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.454980 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f558566_002f_4029_bd8f_d84e571b57eb.slice/crio-bb9cf651977f3c5e99d9581eb878f28ce28c1c74cf4db5e1c09508ec21dc8cb5 WatchSource:0}: Error finding container bb9cf651977f3c5e99d9581eb878f28ce28c1c74cf4db5e1c09508ec21dc8cb5: Status 404 returned error can't find the container with id bb9cf651977f3c5e99d9581eb878f28ce28c1c74cf4db5e1c09508ec21dc8cb5 Nov 03 17:00:01.456350 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:01.456324 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acda7e9_8a14_4301_a64e_ccd1accd8997.slice/crio-b4525d7aa615c8eb85fcd2033ac13732d99ceba1a938e1a9d087d6ac6b7fb771 WatchSource:0}: Error finding container b4525d7aa615c8eb85fcd2033ac13732d99ceba1a938e1a9d087d6ac6b7fb771: Status 404 returned error can't find the container with id b4525d7aa615c8eb85fcd2033ac13732d99ceba1a938e1a9d087d6ac6b7fb771 Nov 03 17:00:01.810096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.810065 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" event={"ID":"0f558566-002f-4029-bd8f-d84e571b57eb","Type":"ContainerStarted","Data":"bb9cf651977f3c5e99d9581eb878f28ce28c1c74cf4db5e1c09508ec21dc8cb5"} Nov 03 17:00:01.812126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.812098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerStarted","Data":"bb1748694fbc2d65106523cfa96e4f1e98bee71d5cb85f5bb30c9dda6ed6fb65"} Nov 03 17:00:01.813601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.813571 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerStarted","Data":"c5258f6829f5c7d43aaeb064cc715021365183a9ee3db29ca8b6d61b15fb9399"} Nov 03 17:00:01.815131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.815108 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerStarted","Data":"f423c6357f5a61a61035c7412eab7b53370b54371a809f0070055aa0f91b5e00"} Nov 03 17:00:01.815233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.815150 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerStarted","Data":"a09e64e5c453778d19e00341f3a0748b6a7ffad23750b4510480e5e597c7c3b9"} Nov 03 17:00:01.816860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.816836 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerStarted","Data":"d8efb206efeb5d48cf2a2fbc9f8171440930ed90099ddd5a2d565b7a0e81574e"} Nov 03 17:00:01.816984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.816890 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerStarted","Data":"3f54ae4e7a520f87388d02203fffb7f98f05b16660e978c9b1f06321eba851c8"} Nov 03 17:00:01.818138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.818115 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerStarted","Data":"7be8aab2f301932789fbd693fbf5d8b7b4856253372e98095f3097daeef4018c"} Nov 03 17:00:01.819480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.819422 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerStarted","Data":"61240c16b1fbf0497a5dcd66182be14cc3d35ca74f4b8ba9546f8be01d6c9b59"} Nov 03 17:00:01.820895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.820876 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" event={"ID":"26130612-35cb-44da-894e-9d243011d96d","Type":"ContainerStarted","Data":"681460a643991366b751ca5129f9afe37481be3b5b5f9e87c28a2cde148f4a06"} Nov 03 17:00:01.822144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.822124 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerStarted","Data":"995df06004f553150245a3670ded19812658f0759018a4b13bfda5914fe363b9"} Nov 03 17:00:01.823328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.823309 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerStarted","Data":"b4525d7aa615c8eb85fcd2033ac13732d99ceba1a938e1a9d087d6ac6b7fb771"} Nov 03 17:00:01.863032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:01.862982 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" podStartSLOduration=1.8629679380000002 podStartE2EDuration="1.862967938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.862035485 +0000 UTC m=+2697.195249075" watchObservedRunningTime="2026-11-03 17:00:01.862967938 +0000 UTC m=+2697.196181506" Nov 03 17:00:02.916849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.912466 6001 generic.go:356] "Generic (PLEG): container finished" podID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerID="d8efb206efeb5d48cf2a2fbc9f8171440930ed90099ddd5a2d565b7a0e81574e" exitCode=60 Nov 03 17:00:02.916849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.912569 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerDied","Data":"d8efb206efeb5d48cf2a2fbc9f8171440930ed90099ddd5a2d565b7a0e81574e"} Nov 03 17:00:02.916849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.913107 6001 scope.go:122] "RemoveContainer" containerID="d8efb206efeb5d48cf2a2fbc9f8171440930ed90099ddd5a2d565b7a0e81574e" Nov 03 17:00:02.918844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.918808 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerStarted","Data":"bddb2d8ae132d34b9599d5d6eb873a39c3a076fd97d74ff6248aa0e71710a2e9"} Nov 03 17:00:02.921298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.921268 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerStarted","Data":"0cdd70a54c8895d428cb33155173c5b7ac5c57d3a5cd69ba0c6da84b39781ad2"} Nov 03 17:00:02.923406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.923375 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" event={"ID":"26130612-35cb-44da-894e-9d243011d96d","Type":"ContainerStarted","Data":"4dff0ce24870387a9b3958fda72c780ab74873256966ba5cd85282304e610aa5"} Nov 03 17:00:02.925330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.925301 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerID="26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd" exitCode=60 Nov 03 17:00:02.925440 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.925379 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerDied","Data":"26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd"} Nov 03 17:00:02.925698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.925680 6001 scope.go:122] "RemoveContainer" containerID="26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd" Nov 03 17:00:02.933654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.933608 6001 generic.go:356] "Generic (PLEG): container finished" podID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerID="2c3d3e4e4cd461289d7199b1fbebb046bccc6715d8bc3e2fc8a80e226b754741" exitCode=60 Nov 03 17:00:02.934084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.934057 6001 scope.go:122] "RemoveContainer" containerID="2c3d3e4e4cd461289d7199b1fbebb046bccc6715d8bc3e2fc8a80e226b754741" Nov 03 17:00:02.934153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.934128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerDied","Data":"2c3d3e4e4cd461289d7199b1fbebb046bccc6715d8bc3e2fc8a80e226b754741"} Nov 03 17:00:02.940729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.940688 6001 generic.go:356] "Generic (PLEG): container finished" podID="0f558566-002f-4029-bd8f-d84e571b57eb" containerID="1fb2435a91408a064ff7808f17fcd2c7c19f2ee3ce813d974271848b241384fb" exitCode=60 Nov 03 17:00:02.941906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.940853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" event={"ID":"0f558566-002f-4029-bd8f-d84e571b57eb","Type":"ContainerDied","Data":"1fb2435a91408a064ff7808f17fcd2c7c19f2ee3ce813d974271848b241384fb"} Nov 03 17:00:02.942681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.942665 6001 scope.go:122] "RemoveContainer" containerID="1fb2435a91408a064ff7808f17fcd2c7c19f2ee3ce813d974271848b241384fb" Nov 03 17:00:02.946817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.945778 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerID="c224ae4a24e19871adc95a1f9a8e7bd289aabee860214beea7a47265c6d3ed48" exitCode=60 Nov 03 17:00:02.946958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.946944 6001 scope.go:122] "RemoveContainer" containerID="c224ae4a24e19871adc95a1f9a8e7bd289aabee860214beea7a47265c6d3ed48" Nov 03 17:00:02.947375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.945804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerDied","Data":"c224ae4a24e19871adc95a1f9a8e7bd289aabee860214beea7a47265c6d3ed48"} Nov 03 17:00:02.949721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.949694 6001 generic.go:356] "Generic (PLEG): container finished" podID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerID="96022b61dd1623dd1cfe64e0a03ae776e97f9cabc3794f35939232c9c43ccc7d" exitCode=60 Nov 03 17:00:02.949954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.949929 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerDied","Data":"96022b61dd1623dd1cfe64e0a03ae776e97f9cabc3794f35939232c9c43ccc7d"} Nov 03 17:00:02.951051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.951015 6001 scope.go:122] "RemoveContainer" containerID="96022b61dd1623dd1cfe64e0a03ae776e97f9cabc3794f35939232c9c43ccc7d" Nov 03 17:00:02.956411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.956380 6001 generic.go:356] "Generic (PLEG): container finished" podID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerID="f423c6357f5a61a61035c7412eab7b53370b54371a809f0070055aa0f91b5e00" exitCode=60 Nov 03 17:00:02.956556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.956456 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerDied","Data":"f423c6357f5a61a61035c7412eab7b53370b54371a809f0070055aa0f91b5e00"} Nov 03 17:00:02.956901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:02.956884 6001 scope.go:122] "RemoveContainer" containerID="f423c6357f5a61a61035c7412eab7b53370b54371a809f0070055aa0f91b5e00" Nov 03 17:00:03.245051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.244997 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" podStartSLOduration=3.244982186 podStartE2EDuration="3.244982186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.091627671 +0000 UTC m=+2698.424841241" watchObservedRunningTime="2026-11-03 17:00:03.244982186 +0000 UTC m=+2698.578195754" Nov 03 17:00:03.458608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.458557 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" podStartSLOduration=3.458542522 podStartE2EDuration="3.458542522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.454712148 +0000 UTC m=+2698.787925717" watchObservedRunningTime="2026-11-03 17:00:03.458542522 +0000 UTC m=+2698.791756091" Nov 03 17:00:03.645688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.642472 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" podStartSLOduration=3.642435963 podStartE2EDuration="3.642435963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:03.636535053 +0000 UTC m=+2698.969748625" watchObservedRunningTime="2026-11-03 17:00:03.642435963 +0000 UTC m=+2698.975649533" Nov 03 17:00:03.977114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.977075 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerStarted","Data":"759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0"} Nov 03 17:00:03.983884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.983071 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerStarted","Data":"3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5"} Nov 03 17:00:03.986099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.985913 6001 generic.go:356] "Generic (PLEG): container finished" podID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerID="bddb2d8ae132d34b9599d5d6eb873a39c3a076fd97d74ff6248aa0e71710a2e9" exitCode=60 Nov 03 17:00:03.986099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.985968 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerDied","Data":"bddb2d8ae132d34b9599d5d6eb873a39c3a076fd97d74ff6248aa0e71710a2e9"} Nov 03 17:00:03.986338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.986286 6001 scope.go:122] "RemoveContainer" containerID="bddb2d8ae132d34b9599d5d6eb873a39c3a076fd97d74ff6248aa0e71710a2e9" Nov 03 17:00:03.990833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.990305 6001 generic.go:356] "Generic (PLEG): container finished" podID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerID="0cdd70a54c8895d428cb33155173c5b7ac5c57d3a5cd69ba0c6da84b39781ad2" exitCode=60 Nov 03 17:00:03.990833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.990393 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerDied","Data":"0cdd70a54c8895d428cb33155173c5b7ac5c57d3a5cd69ba0c6da84b39781ad2"} Nov 03 17:00:03.990974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.990906 6001 scope.go:122] "RemoveContainer" containerID="0cdd70a54c8895d428cb33155173c5b7ac5c57d3a5cd69ba0c6da84b39781ad2" Nov 03 17:00:03.992603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.992581 6001 generic.go:356] "Generic (PLEG): container finished" podID="26130612-35cb-44da-894e-9d243011d96d" containerID="4dff0ce24870387a9b3958fda72c780ab74873256966ba5cd85282304e610aa5" exitCode=60 Nov 03 17:00:03.993106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.992813 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" event={"ID":"26130612-35cb-44da-894e-9d243011d96d","Type":"ContainerDied","Data":"4dff0ce24870387a9b3958fda72c780ab74873256966ba5cd85282304e610aa5"} Nov 03 17:00:03.993106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.992980 6001 scope.go:122] "RemoveContainer" containerID="4dff0ce24870387a9b3958fda72c780ab74873256966ba5cd85282304e610aa5" Nov 03 17:00:03.995533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:03.995510 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerStarted","Data":"e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212"} Nov 03 17:00:04.000838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.000816 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerStarted","Data":"89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762"} Nov 03 17:00:04.015955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.015252 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerStarted","Data":"384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b"} Nov 03 17:00:04.218071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.198840 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" podStartSLOduration=4.19881598 podStartE2EDuration="4.19881598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:04.16371666 +0000 UTC m=+2699.496930234" watchObservedRunningTime="2026-11-03 17:00:04.19881598 +0000 UTC m=+2699.532029550" Nov 03 17:00:04.294695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.287576 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" podStartSLOduration=4.287554716 podStartE2EDuration="4.287554716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:01.913181041 +0000 UTC m=+2697.246394610" watchObservedRunningTime="2026-11-03 17:00:04.287554716 +0000 UTC m=+2699.620768286" Nov 03 17:00:04.704135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.695151 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" podStartSLOduration=4.695114886 podStartE2EDuration="4.695114886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:04.483334615 +0000 UTC m=+2699.816548185" watchObservedRunningTime="2026-11-03 17:00:04.695114886 +0000 UTC m=+2700.028328455" Nov 03 17:00:04.778768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:04.778687 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" podStartSLOduration=4.778666325 podStartE2EDuration="4.778666325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:04.746905896 +0000 UTC m=+2700.080119466" watchObservedRunningTime="2026-11-03 17:00:04.778666325 +0000 UTC m=+2700.111879891" Nov 03 17:00:05.077414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.077354 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" podStartSLOduration=5.077335522 podStartE2EDuration="5.077335522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:05.068278043 +0000 UTC m=+2700.401491615" watchObservedRunningTime="2026-11-03 17:00:05.077335522 +0000 UTC m=+2700.410549105" Nov 03 17:00:05.166099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.166058 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerID="e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212" exitCode=60 Nov 03 17:00:05.166673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.166646 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerDied","Data":"e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212"} Nov 03 17:00:05.166961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.166854 6001 scope.go:122] "RemoveContainer" containerID="26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd" Nov 03 17:00:05.169594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.169423 6001 scope.go:122] "RemoveContainer" containerID="e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212" Nov 03 17:00:05.178491 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.170608 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv_odh-ai-gateway-infra(dbb50354-7d02-488d-b2d5-f44cd56f235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" Nov 03 17:00:05.260055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.260016 6001 generic.go:356] "Generic (PLEG): container finished" podID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerID="89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762" exitCode=60 Nov 03 17:00:05.260228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.260128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerDied","Data":"89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762"} Nov 03 17:00:05.260717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.260658 6001 scope.go:122] "RemoveContainer" containerID="89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762" Nov 03 17:00:05.261102 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.261076 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf_odh-ai-gateway-infra(2acda7e9-8a14-4301-a64e-ccd1accd8997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" Nov 03 17:00:05.273167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.272479 6001 generic.go:356] "Generic (PLEG): container finished" podID="0f558566-002f-4029-bd8f-d84e571b57eb" containerID="c26f8e4f3188e819e4003782662990e9af78f4676c6ce199eaacfcf6b630d0fe" exitCode=60 Nov 03 17:00:05.273167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.272667 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" event={"ID":"0f558566-002f-4029-bd8f-d84e571b57eb","Type":"ContainerDied","Data":"c26f8e4f3188e819e4003782662990e9af78f4676c6ce199eaacfcf6b630d0fe"} Nov 03 17:00:05.273418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.273216 6001 scope.go:122] "RemoveContainer" containerID="c26f8e4f3188e819e4003782662990e9af78f4676c6ce199eaacfcf6b630d0fe" Nov 03 17:00:05.273745 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.273711 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr_odh-ai-gateway-infra(0f558566-002f-4029-bd8f-d84e571b57eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" Nov 03 17:00:05.276030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.275935 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" podStartSLOduration=5.275919035 podStartE2EDuration="5.275919035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:05.268220201 +0000 UTC m=+2700.601433769" watchObservedRunningTime="2026-11-03 17:00:05.275919035 +0000 UTC m=+2700.609132605" Nov 03 17:00:05.277631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.277611 6001 scope.go:122] "RemoveContainer" containerID="2c3d3e4e4cd461289d7199b1fbebb046bccc6715d8bc3e2fc8a80e226b754741" Nov 03 17:00:05.279046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.278990 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerID="384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b" exitCode=60 Nov 03 17:00:05.279157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.279137 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerDied","Data":"384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b"} Nov 03 17:00:05.279931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.279909 6001 scope.go:122] "RemoveContainer" containerID="384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b" Nov 03 17:00:05.282827 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.280364 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq_odh-ai-gateway-infra(1c51ae5a-63de-45e4-ad09-fecfba01f6fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" Nov 03 17:00:05.291879 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.291832 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd\": container with ID starting with 26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd not found: ID does not exist" containerID="26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd" Nov 03 17:00:05.292095 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.292072 6001 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd\": container with ID starting with 26e5a172be1d1f4e8b05e9bec6eb1fd955f1e639d3b7cb85931370c5169fd0cd not found: ID does not exist" Nov 03 17:00:05.292334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.292311 6001 generic.go:356] "Generic (PLEG): container finished" podID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerID="759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0" exitCode=60 Nov 03 17:00:05.292600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.292580 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerDied","Data":"759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0"} Nov 03 17:00:05.293078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.293051 6001 scope.go:122] "RemoveContainer" containerID="759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0" Nov 03 17:00:05.293574 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.293545 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8_odh-ai-gateway-infra(1d017ee1-1f5e-4ebc-814d-72db07b0f010)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" Nov 03 17:00:05.299038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.299012 6001 generic.go:356] "Generic (PLEG): container finished" podID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerID="a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a" exitCode=60 Nov 03 17:00:05.299134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.299088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerDied","Data":"a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a"} Nov 03 17:00:05.299523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.299495 6001 scope.go:122] "RemoveContainer" containerID="a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a" Nov 03 17:00:05.299873 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.299853 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z_odh-ai-gateway-infra(42d1f813-1e5e-4745-9391-dd03d3562c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" Nov 03 17:00:05.302133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.302102 6001 generic.go:356] "Generic (PLEG): container finished" podID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerID="3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5" exitCode=60 Nov 03 17:00:05.302381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.302195 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerDied","Data":"3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5"} Nov 03 17:00:05.302751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.302526 6001 scope.go:122] "RemoveContainer" containerID="3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5" Nov 03 17:00:05.302975 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:05.302953 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9_odh-ai-gateway-infra(9e08b8db-cd60-4eb7-a590-3f3f0c6104ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" Nov 03 17:00:05.312019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.311992 6001 scope.go:122] "RemoveContainer" containerID="1fb2435a91408a064ff7808f17fcd2c7c19f2ee3ce813d974271848b241384fb" Nov 03 17:00:05.336783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.336730 6001 scope.go:122] "RemoveContainer" containerID="c224ae4a24e19871adc95a1f9a8e7bd289aabee860214beea7a47265c6d3ed48" Nov 03 17:00:05.390174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.390103 6001 scope.go:122] "RemoveContainer" containerID="96022b61dd1623dd1cfe64e0a03ae776e97f9cabc3794f35939232c9c43ccc7d" Nov 03 17:00:05.402675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.402645 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:00:05.412560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.412538 6001 scope.go:122] "RemoveContainer" containerID="f423c6357f5a61a61035c7412eab7b53370b54371a809f0070055aa0f91b5e00" Nov 03 17:00:05.430322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:05.430291 6001 scope.go:122] "RemoveContainer" containerID="d8efb206efeb5d48cf2a2fbc9f8171440930ed90099ddd5a2d565b7a0e81574e" Nov 03 17:00:06.294249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.289930 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:06.318933 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.318900 6001 scope.go:122] "RemoveContainer" containerID="3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5" Nov 03 17:00:06.319414 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.319383 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9_odh-ai-gateway-infra(9e08b8db-cd60-4eb7-a590-3f3f0c6104ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" Nov 03 17:00:06.321206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.321172 6001 generic.go:356] "Generic (PLEG): container finished" podID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerID="82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1" exitCode=60 Nov 03 17:00:06.321368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.321345 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerDied","Data":"82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1"} Nov 03 17:00:06.321432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.321407 6001 scope.go:122] "RemoveContainer" containerID="bddb2d8ae132d34b9599d5d6eb873a39c3a076fd97d74ff6248aa0e71710a2e9" Nov 03 17:00:06.321628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.321608 6001 scope.go:122] "RemoveContainer" containerID="82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1" Nov 03 17:00:06.322110 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.322082 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs_odh-ai-gateway-infra(86fc9ebc-6f86-4853-84ab-b09f38817d5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" Nov 03 17:00:06.324607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.324577 6001 generic.go:356] "Generic (PLEG): container finished" podID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerID="1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5" exitCode=60 Nov 03 17:00:06.324688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.324674 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerDied","Data":"1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5"} Nov 03 17:00:06.325148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.325130 6001 scope.go:122] "RemoveContainer" containerID="1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5" Nov 03 17:00:06.325881 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.325599 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw_odh-ai-gateway-infra(2b3d01a4-8458-4bbf-a98e-735627bfad04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" Nov 03 17:00:06.328081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.328057 6001 generic.go:356] "Generic (PLEG): container finished" podID="26130612-35cb-44da-894e-9d243011d96d" containerID="59dd4dccb782a003b6af654133f79a0c6041ea65704a858688ba363a57677d08" exitCode=60 Nov 03 17:00:06.328380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.328366 6001 scope.go:122] "RemoveContainer" containerID="59dd4dccb782a003b6af654133f79a0c6041ea65704a858688ba363a57677d08" Nov 03 17:00:06.328595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.328537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" event={"ID":"26130612-35cb-44da-894e-9d243011d96d","Type":"ContainerDied","Data":"59dd4dccb782a003b6af654133f79a0c6041ea65704a858688ba363a57677d08"} Nov 03 17:00:06.328898 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.328877 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4_odh-ai-gateway-infra(26130612-35cb-44da-894e-9d243011d96d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" podUID="26130612-35cb-44da-894e-9d243011d96d" Nov 03 17:00:06.332006 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.331599 6001 scope.go:122] "RemoveContainer" containerID="e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212" Nov 03 17:00:06.332006 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.331952 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv_odh-ai-gateway-infra(dbb50354-7d02-488d-b2d5-f44cd56f235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" Nov 03 17:00:06.334643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.334614 6001 scope.go:122] "RemoveContainer" containerID="89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762" Nov 03 17:00:06.335118 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.335092 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf_odh-ai-gateway-infra(2acda7e9-8a14-4301-a64e-ccd1accd8997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" Nov 03 17:00:06.375242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.374853 6001 scope.go:122] "RemoveContainer" containerID="0cdd70a54c8895d428cb33155173c5b7ac5c57d3a5cd69ba0c6da84b39781ad2" Nov 03 17:00:06.377127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.377095 6001 scope.go:122] "RemoveContainer" containerID="384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b" Nov 03 17:00:06.377684 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.377653 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq_odh-ai-gateway-infra(1c51ae5a-63de-45e4-ad09-fecfba01f6fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" Nov 03 17:00:06.380924 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.379787 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:06.382121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.382095 6001 scope.go:122] "RemoveContainer" containerID="759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0" Nov 03 17:00:06.382832 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.382798 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8_odh-ai-gateway-infra(1d017ee1-1f5e-4ebc-814d-72db07b0f010)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" Nov 03 17:00:06.402242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:06.402223 6001 scope.go:122] "RemoveContainer" containerID="4dff0ce24870387a9b3958fda72c780ab74873256966ba5cd85282304e610aa5" Nov 03 17:00:06.660569 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:06.657869 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:07.396635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.396603 6001 scope.go:122] "RemoveContainer" containerID="1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5" Nov 03 17:00:07.397102 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:07.397065 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw_odh-ai-gateway-infra(2b3d01a4-8458-4bbf-a98e-735627bfad04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" Nov 03 17:00:07.400946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.400922 6001 scope.go:122] "RemoveContainer" containerID="82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1" Nov 03 17:00:07.401343 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:07.401316 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs_odh-ai-gateway-infra(86fc9ebc-6f86-4853-84ab-b09f38817d5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" Nov 03 17:00:07.731693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.731661 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:07.825379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.825336 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\" (UniqueName: \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\") pod \"26130612-35cb-44da-894e-9d243011d96d\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " Nov 03 17:00:07.825655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.825424 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\") pod \"26130612-35cb-44da-894e-9d243011d96d\" (UID: \"26130612-35cb-44da-894e-9d243011d96d\") " Nov 03 17:00:07.826223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.826152 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca" pod "26130612-35cb-44da-894e-9d243011d96d" (UID: "26130612-35cb-44da-894e-9d243011d96d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:07.826472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.826424 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26130612-35cb-44da-894e-9d243011d96d-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:07.828428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.828393 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b" pod "26130612-35cb-44da-894e-9d243011d96d" (UID: "26130612-35cb-44da-894e-9d243011d96d"). InnerVolumeSpecName "kube-api-access-smr5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:07.927779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:07.927746 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smr5b\" (UniqueName: \"kubernetes.io/projected/26130612-35cb-44da-894e-9d243011d96d-kube-api-access-smr5b\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:08.410603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.410566 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" event={"ID":"26130612-35cb-44da-894e-9d243011d96d","Type":"ContainerDied","Data":"681460a643991366b751ca5129f9afe37481be3b5b5f9e87c28a2cde148f4a06"} Nov 03 17:00:08.411081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.410635 6001 scope.go:122] "RemoveContainer" containerID="59dd4dccb782a003b6af654133f79a0c6041ea65704a858688ba363a57677d08" Nov 03 17:00:08.411081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.410579 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4" Nov 03 17:00:08.521564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.521524 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:08.573023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.572989 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-c1837e-29895420-8rgg4"] Nov 03 17:00:08.791730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:08.791697 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26130612-35cb-44da-894e-9d243011d96d" path="/var/lib/kubelet/pods/26130612-35cb-44da-894e-9d243011d96d/volumes" Nov 03 17:00:10.126843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.126799 6001 scope.go:122] "RemoveContainer" containerID="caee76d9f3c2283bf79aa0ff607929c65fce37baf4930adb608476986d329e53" Nov 03 17:00:10.141703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.141679 6001 scope.go:122] "RemoveContainer" containerID="b5b0dd921d4c9d6e764485ebb5f0bd6aa4b2e883851a870b1a1ec77861d086dd" Nov 03 17:00:10.160716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.155911 6001 scope.go:122] "RemoveContainer" containerID="78373327e0b22ad9d547ce868ab34ba00a3b8f499f9cf01145995d50a4789517" Nov 03 17:00:10.178970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.178943 6001 scope.go:122] "RemoveContainer" containerID="8d4c52a5ee8c2328b206a341ab447eea5b0179359cf103eacb2c914197331273" Nov 03 17:00:10.193519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.193490 6001 scope.go:122] "RemoveContainer" containerID="0070c043facd9a5503e5cb47c2e810048f44b3cdbcffe56e6e0520a57a709aef" Nov 03 17:00:10.205077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.205053 6001 scope.go:122] "RemoveContainer" containerID="74942f4f01e1cef36b567717875b95196008aea98786e9ad8265f30064b8332d" Nov 03 17:00:10.216497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.216473 6001 scope.go:122] "RemoveContainer" containerID="f486cb759d473d119a6fb7dccdec17685076fb5ea6332342ebb20650baa24eb5" Nov 03 17:00:10.228608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.228585 6001 scope.go:122] "RemoveContainer" containerID="d952af9d128aff326975c1cc8efff2a760d8c8e72f4dfbe5031de0b6b7accf56" Nov 03 17:00:10.239375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.239321 6001 scope.go:122] "RemoveContainer" containerID="10caff164cbd8f6b24f778c3fc022f918a158bf7ed3e07a9c1e1df2a85d42f57" Nov 03 17:00:10.516139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.516084 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf"] Nov 03 17:00:10.517034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517009 6001 cpu_manager.go:414] "containerMap: removing container" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.517034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517033 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.517258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517087 6001 cpu_manager.go:414] "containerMap: removing container" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.517258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517096 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.517359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517329 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.517407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.517360 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="26130612-35cb-44da-894e-9d243011d96d" containerName="cleanup" Nov 03 17:00:10.523479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.523420 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.578525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.576778 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:10.578525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.577319 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf"] Nov 03 17:00:10.679425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.679322 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85smn\" (UniqueName: \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.679638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.679453 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.679772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.679752 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.781975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.781939 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.782173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.782156 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.782367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.782350 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85smn\" (UniqueName: \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.782705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.782677 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.784844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.784812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.807877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.807842 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85smn\" (UniqueName: \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\") pod \"maas-api-revoke-sub-526262a1-k6wvf\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.838124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:10.838085 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:10.987844 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:10.987784 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:10.988572 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:10.988538 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:11.157789 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:11.157750 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd2d81a_7dff_47c5_a266_dd72cd6768c7.slice/crio-f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd WatchSource:0}: Error finding container f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd: Status 404 returned error can't find the container with id f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd Nov 03 17:00:11.160206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:11.160180 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf"] Nov 03 17:00:11.459531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:11.459489 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" event={"ID":"efd2d81a-7dff-47c5-a266-dd72cd6768c7","Type":"ContainerStarted","Data":"f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd"} Nov 03 17:00:12.472994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:12.472947 6001 generic.go:356] "Generic (PLEG): container finished" podID="efd2d81a-7dff-47c5-a266-dd72cd6768c7" containerID="e0ce659577b5fc538e117c42387f0a1668cfe02b7788a46652500c3ed5f37933" exitCode=0 Nov 03 17:00:12.473410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:12.473032 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" event={"ID":"efd2d81a-7dff-47c5-a266-dd72cd6768c7","Type":"ContainerDied","Data":"e0ce659577b5fc538e117c42387f0a1668cfe02b7788a46652500c3ed5f37933"} Nov 03 17:00:12.775009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:12.772575 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 17:00:12.775009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:12.772865 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" containerID="cri-o://07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c" gracePeriod=30 Nov 03 17:00:13.891739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.891650 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:13.965910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.965873 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\" (UniqueName: \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\") pod \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " Nov 03 17:00:13.966106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.966057 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\") pod \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " Nov 03 17:00:13.966293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.966270 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\") pod \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\" (UID: \"efd2d81a-7dff-47c5-a266-dd72cd6768c7\") " Nov 03 17:00:13.966779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.966748 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca" pod "efd2d81a-7dff-47c5-a266-dd72cd6768c7" (UID: "efd2d81a-7dff-47c5-a266-dd72cd6768c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:13.967163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.967140 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:13.968403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.968368 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca" pod "efd2d81a-7dff-47c5-a266-dd72cd6768c7" (UID: "efd2d81a-7dff-47c5-a266-dd72cd6768c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:13.969554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:13.969527 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn" pod "efd2d81a-7dff-47c5-a266-dd72cd6768c7" (UID: "efd2d81a-7dff-47c5-a266-dd72cd6768c7"). InnerVolumeSpecName "kube-api-access-85smn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:14.084745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:14.084709 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd2d81a-7dff-47c5-a266-dd72cd6768c7-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:14.084745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:14.084743 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85smn\" (UniqueName: \"kubernetes.io/projected/efd2d81a-7dff-47c5-a266-dd72cd6768c7-kube-api-access-85smn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:14.492215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:14.492178 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" event={"ID":"efd2d81a-7dff-47c5-a266-dd72cd6768c7","Type":"ContainerDied","Data":"f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd"} Nov 03 17:00:14.492413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:14.492233 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f30226b2330e28073d0013562a50af8a185c10ef8b2c8fe82ee2b8fb62e310bd" Nov 03 17:00:14.492413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:14.492186 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf" Nov 03 17:00:15.781832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:15.781789 6001 scope.go:122] "RemoveContainer" containerID="759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0" Nov 03 17:00:15.782434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:15.781887 6001 scope.go:122] "RemoveContainer" containerID="a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a" Nov 03 17:00:15.782434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:15.782142 6001 scope.go:122] "RemoveContainer" containerID="e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212" Nov 03 17:00:15.782658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:15.782490 6001 scope.go:122] "RemoveContainer" containerID="3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5" Nov 03 17:00:16.497814 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:16.497783 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:16.509027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.509003 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerStarted","Data":"4de6f751f434e0e62237ddd280bcfac8f1ab29132f640542f40bdc356f615687"} Nov 03 17:00:16.511158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.511132 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerStarted","Data":"703b4ee6746d1a6da1a1b72c20eb6f10d0f33bb877f51f6e7332060cff907f68"} Nov 03 17:00:16.513049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.513027 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerStarted","Data":"ef668bda7debc5a4338aaace344efbf94494564cf2c85deb3880de0dd057289f"} Nov 03 17:00:16.514842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.514821 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerStarted","Data":"621142fb896eb72894d9599d8c078b25d4785b056e3fee45c5939d0430273a94"} Nov 03 17:00:16.792458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.792421 6001 scope.go:122] "RemoveContainer" containerID="89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762" Nov 03 17:00:16.792950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.792606 6001 scope.go:122] "RemoveContainer" containerID="c26f8e4f3188e819e4003782662990e9af78f4676c6ce199eaacfcf6b630d0fe" Nov 03 17:00:16.877954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:16.876414 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:17.569471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.567421 6001 generic.go:356] "Generic (PLEG): container finished" podID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerID="703b4ee6746d1a6da1a1b72c20eb6f10d0f33bb877f51f6e7332060cff907f68" exitCode=60 Nov 03 17:00:17.569471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.567540 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerDied","Data":"703b4ee6746d1a6da1a1b72c20eb6f10d0f33bb877f51f6e7332060cff907f68"} Nov 03 17:00:17.569471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.567602 6001 scope.go:122] "RemoveContainer" containerID="759a4ed80eac6ae3071a040a38c9f8ee49edc6c1e26a3a42bcf162376c97f0c0" Nov 03 17:00:17.569471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.568136 6001 scope.go:122] "RemoveContainer" containerID="703b4ee6746d1a6da1a1b72c20eb6f10d0f33bb877f51f6e7332060cff907f68" Nov 03 17:00:17.569471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:17.568592 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8_odh-ai-gateway-infra(1d017ee1-1f5e-4ebc-814d-72db07b0f010)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" Nov 03 17:00:17.580966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.580926 6001 generic.go:356] "Generic (PLEG): container finished" podID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerID="ef668bda7debc5a4338aaace344efbf94494564cf2c85deb3880de0dd057289f" exitCode=60 Nov 03 17:00:17.581141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.581044 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerDied","Data":"ef668bda7debc5a4338aaace344efbf94494564cf2c85deb3880de0dd057289f"} Nov 03 17:00:17.598200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.598164 6001 generic.go:356] "Generic (PLEG): container finished" podID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerID="621142fb896eb72894d9599d8c078b25d4785b056e3fee45c5939d0430273a94" exitCode=60 Nov 03 17:00:17.598589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.598563 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerDied","Data":"621142fb896eb72894d9599d8c078b25d4785b056e3fee45c5939d0430273a94"} Nov 03 17:00:17.599225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.599205 6001 scope.go:122] "RemoveContainer" containerID="621142fb896eb72894d9599d8c078b25d4785b056e3fee45c5939d0430273a94" Nov 03 17:00:17.599821 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:17.599795 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9_odh-ai-gateway-infra(9e08b8db-cd60-4eb7-a590-3f3f0c6104ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" Nov 03 17:00:17.604834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.603761 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerID="4de6f751f434e0e62237ddd280bcfac8f1ab29132f640542f40bdc356f615687" exitCode=60 Nov 03 17:00:17.604834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.603866 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerDied","Data":"4de6f751f434e0e62237ddd280bcfac8f1ab29132f640542f40bdc356f615687"} Nov 03 17:00:17.604834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.604328 6001 scope.go:122] "RemoveContainer" containerID="4de6f751f434e0e62237ddd280bcfac8f1ab29132f640542f40bdc356f615687" Nov 03 17:00:17.604834 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:17.604776 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv_odh-ai-gateway-infra(dbb50354-7d02-488d-b2d5-f44cd56f235c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" Nov 03 17:00:17.609016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.608807 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:17.614712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.614684 6001 scope.go:122] "RemoveContainer" containerID="a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a" Nov 03 17:00:17.656558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.656531 6001 scope.go:122] "RemoveContainer" containerID="3c7e40681f39a61ff22d32bdd98fa73a629314046e00ec4dfeb0c64a189d73c5" Nov 03 17:00:17.675064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.675035 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:17.682485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.681762 6001 scope.go:122] "RemoveContainer" containerID="e8f00de64804a82e0fd17ffc065ccb2cfe844b0a8efd253ce1b00c533cfb0212" Nov 03 17:00:17.727016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.726986 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:17.786816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.783558 6001 scope.go:122] "RemoveContainer" containerID="384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b" Nov 03 17:00:17.976468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.968085 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:17.976468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.971198 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\") pod \"42d1f813-1e5e-4745-9391-dd03d3562c78\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " Nov 03 17:00:17.976468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.971245 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\" (UniqueName: \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\") pod \"42d1f813-1e5e-4745-9391-dd03d3562c78\" (UID: \"42d1f813-1e5e-4745-9391-dd03d3562c78\") " Nov 03 17:00:17.976468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.971621 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca" pod "42d1f813-1e5e-4745-9391-dd03d3562c78" (UID: "42d1f813-1e5e-4745-9391-dd03d3562c78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:17.976468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.973016 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42d1f813-1e5e-4745-9391-dd03d3562c78-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:17.995105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:17.994597 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2" pod "42d1f813-1e5e-4745-9391-dd03d3562c78" (UID: "42d1f813-1e5e-4745-9391-dd03d3562c78"). InnerVolumeSpecName "kube-api-access-gj9v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.074819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.074788 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9v2\" (UniqueName: \"kubernetes.io/projected/42d1f813-1e5e-4745-9391-dd03d3562c78-kube-api-access-gj9v2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.251637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.251615 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 17:00:18.276996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.276956 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277057 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277211 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277302 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277401 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:18.277478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277458 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:18.277613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277479 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277535 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277624 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.277718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277631 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:18.277718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277648 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:18.277718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.277698 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\") pod \"244a6ce2-1c95-4b87-a843-32fc0a178838\" (UID: \"244a6ce2-1c95-4b87-a843-32fc0a178838\") " Nov 03 17:00:18.278189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278163 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.278634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278599 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/244a6ce2-1c95-4b87-a843-32fc0a178838-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.278744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278636 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.278744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278653 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.278744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278667 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.278744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.278682 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.279936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.279905 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "kube-api-access-6vhr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.280053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.280015 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:18.280110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.280048 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:18.280110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.280093 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token" pod "244a6ce2-1c95-4b87-a843-32fc0a178838" (UID: "244a6ce2-1c95-4b87-a843-32fc0a178838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.380563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.380530 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.380563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.380563 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.380783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.380578 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhr8\" (UniqueName: \"kubernetes.io/projected/244a6ce2-1c95-4b87-a843-32fc0a178838-kube-api-access-6vhr8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.380783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.380591 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/244a6ce2-1c95-4b87-a843-32fc0a178838-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.522643 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k"] Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523841 6001 cpu_manager.go:414] "containerMap: removing container" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523866 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523914 6001 cpu_manager.go:414] "containerMap: removing container" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523922 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523943 6001 cpu_manager.go:414] "containerMap: removing container" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523951 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523992 6001 cpu_manager.go:414] "containerMap: removing container" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.523998 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524019 6001 cpu_manager.go:414] "containerMap: removing container" podUID="efd2d81a-7dff-47c5-a266-dd72cd6768c7" containerName="revoke-keys" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524025 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd2d81a-7dff-47c5-a266-dd72cd6768c7" containerName="revoke-keys" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524356 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524381 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerName="istio-proxy" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524397 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524410 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:18.531483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.524425 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd2d81a-7dff-47c5-a266-dd72cd6768c7" containerName="revoke-keys" Nov 03 17:00:18.543148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.536844 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.549730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.549691 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k"] Nov 03 17:00:18.561724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.561696 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:00:18.598501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.598460 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.598809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.598783 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzvg6\" (UniqueName: \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.598896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.598834 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.619496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.619466 6001 generic.go:356] "Generic (PLEG): container finished" podID="244a6ce2-1c95-4b87-a843-32fc0a178838" containerID="07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c" exitCode=0 Nov 03 17:00:18.619629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.619519 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" Nov 03 17:00:18.619629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.619568 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerDied","Data":"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c"} Nov 03 17:00:18.619749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.619633 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97" event={"ID":"244a6ce2-1c95-4b87-a843-32fc0a178838","Type":"ContainerDied","Data":"13c8f3b7fc602fec37f3b8f447d367ad24b16a8b68dd6c594f6425b9f2a1ca88"} Nov 03 17:00:18.619749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.619669 6001 scope.go:122] "RemoveContainer" containerID="07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c" Nov 03 17:00:18.621977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.621940 6001 generic.go:356] "Generic (PLEG): container finished" podID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerID="92fecd4f0943dbbdeb32d2022da1b8319a9d0cf99a066a7711ee770dc2cd9e57" exitCode=60 Nov 03 17:00:18.622077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.622002 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerDied","Data":"92fecd4f0943dbbdeb32d2022da1b8319a9d0cf99a066a7711ee770dc2cd9e57"} Nov 03 17:00:18.622619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.622590 6001 scope.go:122] "RemoveContainer" containerID="92fecd4f0943dbbdeb32d2022da1b8319a9d0cf99a066a7711ee770dc2cd9e57" Nov 03 17:00:18.623054 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:18.623014 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf_odh-ai-gateway-infra(2acda7e9-8a14-4301-a64e-ccd1accd8997)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" Nov 03 17:00:18.624878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.624855 6001 generic.go:356] "Generic (PLEG): container finished" podID="0f558566-002f-4029-bd8f-d84e571b57eb" containerID="80904ac2782d5f3b987edcff908438d523e841ce28f12d93984746b23343e333" exitCode=60 Nov 03 17:00:18.624990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.624925 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" event={"ID":"0f558566-002f-4029-bd8f-d84e571b57eb","Type":"ContainerDied","Data":"80904ac2782d5f3b987edcff908438d523e841ce28f12d93984746b23343e333"} Nov 03 17:00:18.625378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.625362 6001 scope.go:122] "RemoveContainer" containerID="80904ac2782d5f3b987edcff908438d523e841ce28f12d93984746b23343e333" Nov 03 17:00:18.625770 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:18.625741 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr_odh-ai-gateway-infra(0f558566-002f-4029-bd8f-d84e571b57eb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" Nov 03 17:00:18.627111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.627089 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerStarted","Data":"f7ad0680a1702a8d922cb359b0c71cc950e9bfacb56145bc9f5fa4223900c420"} Nov 03 17:00:18.632957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.632724 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" event={"ID":"42d1f813-1e5e-4745-9391-dd03d3562c78","Type":"ContainerDied","Data":"a09e64e5c453778d19e00341f3a0748b6a7ffad23750b4510480e5e597c7c3b9"} Nov 03 17:00:18.632957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.632741 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z" Nov 03 17:00:18.634855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.634833 6001 scope.go:122] "RemoveContainer" containerID="aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9" Nov 03 17:00:18.658373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.658026 6001 scope.go:122] "RemoveContainer" containerID="07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c" Nov 03 17:00:18.658997 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:18.658672 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c\": container with ID starting with 07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c not found: ID does not exist" containerID="07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c" Nov 03 17:00:18.658997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.658714 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c"} err="failed to get container status \"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c\": rpc error: code = NotFound desc = could not find container \"07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c\": container with ID starting with 07bf23d6d8df69a4e7750c60f53ccbc53deb5cc5f3dbd9d233e1abec0ec01d2c not found: ID does not exist" Nov 03 17:00:18.658997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.658743 6001 scope.go:122] "RemoveContainer" containerID="aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9" Nov 03 17:00:18.659663 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:18.659282 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9\": container with ID starting with aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9 not found: ID does not exist" containerID="aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9" Nov 03 17:00:18.659663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.659314 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9"} err="failed to get container status \"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9\": rpc error: code = NotFound desc = could not find container \"aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9\": container with ID starting with aea9b3886736bcf7cfaa18c61ecd510860d1522c90e348bd749a455f0b8576d9 not found: ID does not exist" Nov 03 17:00:18.659663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.659339 6001 scope.go:122] "RemoveContainer" containerID="89f87d264d40ef16f9c6b50676fdc3cde78d5196b78259f493689a6625f54762" Nov 03 17:00:18.686816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.686719 6001 scope.go:122] "RemoveContainer" containerID="c26f8e4f3188e819e4003782662990e9af78f4676c6ce199eaacfcf6b630d0fe" Nov 03 17:00:18.700647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.700615 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.701726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.701596 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.703358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.703335 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzvg6\" (UniqueName: \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.703542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.703523 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.706565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.706543 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.709595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.709498 6001 scope.go:122] "RemoveContainer" containerID="ef668bda7debc5a4338aaace344efbf94494564cf2c85deb3880de0dd057289f" Nov 03 17:00:18.748832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.748753 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzvg6\" (UniqueName: \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\") pod \"maas-api-revoke-sub-dff75263-skr8k\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.791377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.791120 6001 scope.go:122] "RemoveContainer" containerID="82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1" Nov 03 17:00:18.819723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.819658 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:18.889991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.889731 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:18.900062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.899998 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:18.914583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.911067 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\" (UniqueName: \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\") pod \"dbb50354-7d02-488d-b2d5-f44cd56f235c\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " Nov 03 17:00:18.914583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.911175 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\") pod \"dbb50354-7d02-488d-b2d5-f44cd56f235c\" (UID: \"dbb50354-7d02-488d-b2d5-f44cd56f235c\") " Nov 03 17:00:18.914583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.914546 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca" pod "dbb50354-7d02-488d-b2d5-f44cd56f235c" (UID: "dbb50354-7d02-488d-b2d5-f44cd56f235c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:18.925690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.925630 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7" pod "dbb50354-7d02-488d-b2d5-f44cd56f235c" (UID: "dbb50354-7d02-488d-b2d5-f44cd56f235c"). InnerVolumeSpecName "kube-api-access-twxz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:18.928806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.928770 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895420-t2g6z"] Nov 03 17:00:18.981873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.981842 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:18.989499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:18.988051 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:19.014017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.013966 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\") pod \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " Nov 03 17:00:19.014194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.014137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\") pod \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " Nov 03 17:00:19.014268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.014238 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\" (UniqueName: \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\") pod \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\" (UID: \"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba\") " Nov 03 17:00:19.014333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.014321 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\" (UniqueName: \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\") pod \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\" (UID: \"1d017ee1-1f5e-4ebc-814d-72db07b0f010\") " Nov 03 17:00:19.014658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.014616 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca" pod "9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" (UID: "9e08b8db-cd60-4eb7-a590-3f3f0c6104ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:19.015080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.015055 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxz7\" (UniqueName: \"kubernetes.io/projected/dbb50354-7d02-488d-b2d5-f44cd56f235c-kube-api-access-twxz7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.015195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.015086 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.015195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.015101 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb50354-7d02-488d-b2d5-f44cd56f235c-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.015195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.015169 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca" pod "1d017ee1-1f5e-4ebc-814d-72db07b0f010" (UID: "1d017ee1-1f5e-4ebc-814d-72db07b0f010"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:19.018369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.018340 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56" pod "1d017ee1-1f5e-4ebc-814d-72db07b0f010" (UID: "1d017ee1-1f5e-4ebc-814d-72db07b0f010"). InnerVolumeSpecName "kube-api-access-rpg56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:19.018501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.018379 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw" pod "9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" (UID: "9e08b8db-cd60-4eb7-a590-3f3f0c6104ba"). InnerVolumeSpecName "kube-api-access-dgdmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:19.043806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.043724 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 17:00:19.053627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.053604 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c1837e-openshift-default-6dbf9fd65c-ttd97"] Nov 03 17:00:19.116602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.116573 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgdmw\" (UniqueName: \"kubernetes.io/projected/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba-kube-api-access-dgdmw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.116781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.116770 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpg56\" (UniqueName: \"kubernetes.io/projected/1d017ee1-1f5e-4ebc-814d-72db07b0f010-kube-api-access-rpg56\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.116875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.116866 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d017ee1-1f5e-4ebc-814d-72db07b0f010-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:19.159607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.159578 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:00:19.159767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.159625 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:00:19.328157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.328091 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k"] Nov 03 17:00:19.646855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.646784 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" Nov 03 17:00:19.647891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.647515 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8" event={"ID":"1d017ee1-1f5e-4ebc-814d-72db07b0f010","Type":"ContainerDied","Data":"c5258f6829f5c7d43aaeb064cc715021365183a9ee3db29ca8b6d61b15fb9399"} Nov 03 17:00:19.647891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.647597 6001 scope.go:122] "RemoveContainer" containerID="703b4ee6746d1a6da1a1b72c20eb6f10d0f33bb877f51f6e7332060cff907f68" Nov 03 17:00:19.651024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.651003 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" Nov 03 17:00:19.651159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.651013 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9" event={"ID":"9e08b8db-cd60-4eb7-a590-3f3f0c6104ba","Type":"ContainerDied","Data":"3f54ae4e7a520f87388d02203fffb7f98f05b16660e978c9b1f06321eba851c8"} Nov 03 17:00:19.654818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.653876 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerStarted","Data":"13908deea35ad123875daf7d4994dd2dd821c4a0c378ebcee51d005ec1b7a8e1"} Nov 03 17:00:19.657085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.657058 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" Nov 03 17:00:19.657233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.657053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv" event={"ID":"dbb50354-7d02-488d-b2d5-f44cd56f235c","Type":"ContainerDied","Data":"995df06004f553150245a3670ded19812658f0759018a4b13bfda5914fe363b9"} Nov 03 17:00:19.662911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.662891 6001 scope.go:122] "RemoveContainer" containerID="621142fb896eb72894d9599d8c078b25d4785b056e3fee45c5939d0430273a94" Nov 03 17:00:19.667001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.666970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" event={"ID":"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1","Type":"ContainerStarted","Data":"32f438e517ef2a0b96314f452244ec2b6a7017838691081d5f0cd0b1692d2138"} Nov 03 17:00:19.669210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.669182 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerID="f7ad0680a1702a8d922cb359b0c71cc950e9bfacb56145bc9f5fa4223900c420" exitCode=60 Nov 03 17:00:19.669318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.669249 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerDied","Data":"f7ad0680a1702a8d922cb359b0c71cc950e9bfacb56145bc9f5fa4223900c420"} Nov 03 17:00:19.669892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.669871 6001 scope.go:122] "RemoveContainer" containerID="f7ad0680a1702a8d922cb359b0c71cc950e9bfacb56145bc9f5fa4223900c420" Nov 03 17:00:19.670339 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:19.670305 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq_odh-ai-gateway-infra(1c51ae5a-63de-45e4-ad09-fecfba01f6fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" Nov 03 17:00:19.676323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.676301 6001 scope.go:122] "RemoveContainer" containerID="4de6f751f434e0e62237ddd280bcfac8f1ab29132f640542f40bdc356f615687" Nov 03 17:00:19.689141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.689113 6001 scope.go:122] "RemoveContainer" containerID="384052f1d2d8f20222d407f88129af56114a882ec4b20761b1869621fd0d8a2b" Nov 03 17:00:19.734715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.734675 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:19.783058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.782392 6001 scope.go:122] "RemoveContainer" containerID="1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5" Nov 03 17:00:19.789510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.789485 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:19.796000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.795974 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895420-dxxh9"] Nov 03 17:00:19.802531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.802502 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:19.852098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.852070 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:19.875879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.875838 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:19.882267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.882206 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895420-6qfjv"] Nov 03 17:00:19.925844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.925753 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:19.928645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:19.928609 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895420-5wkt8"] Nov 03 17:00:20.230309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.230275 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 17:00:20.231723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.231687 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" containerID="cri-o://02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec" gracePeriod=30 Nov 03 17:00:20.240938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.239660 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 17:00:20.240938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.239889 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" containerID="cri-o://58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca" gracePeriod=30 Nov 03 17:00:20.243078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.243048 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 17:00:20.243257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.243236 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" containerID="cri-o://09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1" gracePeriod=30 Nov 03 17:00:20.279793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.279718 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:20.356271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.356240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\" (UniqueName: \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\") pod \"0f558566-002f-4029-bd8f-d84e571b57eb\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " Nov 03 17:00:20.356498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.356365 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\") pod \"0f558566-002f-4029-bd8f-d84e571b57eb\" (UID: \"0f558566-002f-4029-bd8f-d84e571b57eb\") " Nov 03 17:00:20.357254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.357226 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca" pod "0f558566-002f-4029-bd8f-d84e571b57eb" (UID: "0f558566-002f-4029-bd8f-d84e571b57eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:20.359740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.359713 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw" pod "0f558566-002f-4029-bd8f-d84e571b57eb" (UID: "0f558566-002f-4029-bd8f-d84e571b57eb"). InnerVolumeSpecName "kube-api-access-5lgxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:20.366874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.366846 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:20.460812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.460626 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\" (UniqueName: \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\") pod \"2acda7e9-8a14-4301-a64e-ccd1accd8997\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " Nov 03 17:00:20.461009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.460922 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\") pod \"2acda7e9-8a14-4301-a64e-ccd1accd8997\" (UID: \"2acda7e9-8a14-4301-a64e-ccd1accd8997\") " Nov 03 17:00:20.461330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.461277 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca" pod "2acda7e9-8a14-4301-a64e-ccd1accd8997" (UID: "2acda7e9-8a14-4301-a64e-ccd1accd8997"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:20.461758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.461737 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2acda7e9-8a14-4301-a64e-ccd1accd8997-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.461845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.461764 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lgxw\" (UniqueName: \"kubernetes.io/projected/0f558566-002f-4029-bd8f-d84e571b57eb-kube-api-access-5lgxw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.461845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.461778 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f558566-002f-4029-bd8f-d84e571b57eb-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.464799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.464767 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9" pod "2acda7e9-8a14-4301-a64e-ccd1accd8997" (UID: "2acda7e9-8a14-4301-a64e-ccd1accd8997"). InnerVolumeSpecName "kube-api-access-vcdj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:20.564114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.564076 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcdj9\" (UniqueName: \"kubernetes.io/projected/2acda7e9-8a14-4301-a64e-ccd1accd8997-kube-api-access-vcdj9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:20.696535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.695965 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" Nov 03 17:00:20.696535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.695988 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf" event={"ID":"2acda7e9-8a14-4301-a64e-ccd1accd8997","Type":"ContainerDied","Data":"b4525d7aa615c8eb85fcd2033ac13732d99ceba1a938e1a9d087d6ac6b7fb771"} Nov 03 17:00:20.696535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.696105 6001 scope.go:122] "RemoveContainer" containerID="92fecd4f0943dbbdeb32d2022da1b8319a9d0cf99a066a7711ee770dc2cd9e57" Nov 03 17:00:20.699291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.699264 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" Nov 03 17:00:20.709506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.702535 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr" event={"ID":"0f558566-002f-4029-bd8f-d84e571b57eb","Type":"ContainerDied","Data":"bb9cf651977f3c5e99d9581eb878f28ce28c1c74cf4db5e1c09508ec21dc8cb5"} Nov 03 17:00:20.714008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.710190 6001 scope.go:122] "RemoveContainer" containerID="80904ac2782d5f3b987edcff908438d523e841ce28f12d93984746b23343e333" Nov 03 17:00:20.714008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.711569 6001 generic.go:356] "Generic (PLEG): container finished" podID="8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" containerID="f61da15aabc4f8c2c513d391ae05305ed193fe11f520ab9ae148d8f37f2092ac" exitCode=0 Nov 03 17:00:20.714008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.711705 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" event={"ID":"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1","Type":"ContainerDied","Data":"f61da15aabc4f8c2c513d391ae05305ed193fe11f520ab9ae148d8f37f2092ac"} Nov 03 17:00:20.722158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.722123 6001 generic.go:356] "Generic (PLEG): container finished" podID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerID="13908deea35ad123875daf7d4994dd2dd821c4a0c378ebcee51d005ec1b7a8e1" exitCode=60 Nov 03 17:00:20.722304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.722282 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerDied","Data":"13908deea35ad123875daf7d4994dd2dd821c4a0c378ebcee51d005ec1b7a8e1"} Nov 03 17:00:20.722788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.722769 6001 scope.go:122] "RemoveContainer" containerID="13908deea35ad123875daf7d4994dd2dd821c4a0c378ebcee51d005ec1b7a8e1" Nov 03 17:00:20.723226 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:20.723202 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs_odh-ai-gateway-infra(86fc9ebc-6f86-4853-84ab-b09f38817d5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" Nov 03 17:00:20.725052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.725030 6001 scope.go:122] "RemoveContainer" containerID="82e2d3c48300c56afdd2180d16c986ddbaa529d75f0281abea5285fa50eef8f1" Nov 03 17:00:20.726414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.726354 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerStarted","Data":"a69fbc269e8d944e23b5a69faa9d6c995295f10f2a52d47e78b8bbbcd4937384"} Nov 03 17:00:20.740934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.740898 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:20.813711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.813673 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" path="/var/lib/kubelet/pods/1d017ee1-1f5e-4ebc-814d-72db07b0f010/volumes" Nov 03 17:00:20.814823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.814798 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244a6ce2-1c95-4b87-a843-32fc0a178838" path="/var/lib/kubelet/pods/244a6ce2-1c95-4b87-a843-32fc0a178838/volumes" Nov 03 17:00:20.815608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.815589 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" path="/var/lib/kubelet/pods/42d1f813-1e5e-4745-9391-dd03d3562c78/volumes" Nov 03 17:00:20.820094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.816757 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" path="/var/lib/kubelet/pods/9e08b8db-cd60-4eb7-a590-3f3f0c6104ba/volumes" Nov 03 17:00:20.820094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.817384 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" path="/var/lib/kubelet/pods/dbb50354-7d02-488d-b2d5-f44cd56f235c/volumes" Nov 03 17:00:20.926823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.926789 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd"] Nov 03 17:00:20.927996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.927975 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928001 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928018 6001 cpu_manager.go:414] "containerMap: removing container" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:20.928048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928026 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928069 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928078 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928088 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928095 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928108 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928115 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928124 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928131 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928144 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928151 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928172 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928179 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928203 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928209 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928232 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928238 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928248 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928256 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928268 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928274 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928298 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928304 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928315 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928321 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928333 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928339 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.928507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928513 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928525 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928534 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928541 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928547 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928556 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928562 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928568 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb50354-7d02-488d-b2d5-f44cd56f235c" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928574 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928579 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d1f813-1e5e-4745-9391-dd03d3562c78" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928592 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928597 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" containerName="cleanup" Nov 03 17:00:20.929013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.928604 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:20.932780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.932759 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:20.935729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.935702 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:20.943570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.943549 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895420-ncgrr"] Nov 03 17:00:20.960573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.960509 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd"] Nov 03 17:00:20.972315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.972282 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cmhh\" (UniqueName: \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:20.972694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.972669 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:20.972783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.972769 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:20.991779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:20.991752 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:21.031256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.031200 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:21.035236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.035208 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895420-bk9lf"] Nov 03 17:00:21.073860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.073826 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\") pod \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " Nov 03 17:00:21.074121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.073876 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\" (UniqueName: \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\") pod \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\" (UID: \"1c51ae5a-63de-45e4-ad09-fecfba01f6fa\") " Nov 03 17:00:21.074331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.074307 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.074432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.074354 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca" pod "1c51ae5a-63de-45e4-ad09-fecfba01f6fa" (UID: "1c51ae5a-63de-45e4-ad09-fecfba01f6fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:21.074432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.074416 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.074593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.074556 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cmhh\" (UniqueName: \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.075000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.074976 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:21.075397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.075346 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.076932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.076910 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.077032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.077000 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z" pod "1c51ae5a-63de-45e4-ad09-fecfba01f6fa" (UID: "1c51ae5a-63de-45e4-ad09-fecfba01f6fa"). InnerVolumeSpecName "kube-api-access-sp59z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:21.101696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.101661 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cmhh\" (UniqueName: \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\") pod \"maas-api-revoke-sub-27b78354-6f8hd\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.176543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.176505 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp59z\" (UniqueName: \"kubernetes.io/projected/1c51ae5a-63de-45e4-ad09-fecfba01f6fa-kube-api-access-sp59z\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:21.307966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.307876 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:21.628868 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:21.628829 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed72ab3_ee6a_43f9_a671_4206ef791512.slice/crio-3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa WatchSource:0}: Error finding container 3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa: Status 404 returned error can't find the container with id 3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa Nov 03 17:00:21.629533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.629364 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd"] Nov 03 17:00:21.714221 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:21.714177 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:21.737681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.737645 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" event={"ID":"2ed72ab3-ee6a-43f9-a671-4206ef791512","Type":"ContainerStarted","Data":"3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa"} Nov 03 17:00:21.739001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.738979 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" Nov 03 17:00:21.739076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.739013 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq" event={"ID":"1c51ae5a-63de-45e4-ad09-fecfba01f6fa","Type":"ContainerDied","Data":"bb1748694fbc2d65106523cfa96e4f1e98bee71d5cb85f5bb30c9dda6ed6fb65"} Nov 03 17:00:21.739146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.739092 6001 scope.go:122] "RemoveContainer" containerID="f7ad0680a1702a8d922cb359b0c71cc950e9bfacb56145bc9f5fa4223900c420" Nov 03 17:00:21.740760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.740737 6001 generic.go:356] "Generic (PLEG): container finished" podID="3844c53f-0103-4055-8a03-9416753e16e4" containerID="0e354216adbad57e9e1302256588b77aeb4ebb787a327b6e384e3706d472ae56" exitCode=0 Nov 03 17:00:21.740853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.740811 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerDied","Data":"0e354216adbad57e9e1302256588b77aeb4ebb787a327b6e384e3706d472ae56"} Nov 03 17:00:21.741381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.741359 6001 scope.go:122] "RemoveContainer" containerID="0e354216adbad57e9e1302256588b77aeb4ebb787a327b6e384e3706d472ae56" Nov 03 17:00:21.743848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.743349 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:21.743848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.743464 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:21.743848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.743517 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:21.746914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.746888 6001 generic.go:356] "Generic (PLEG): container finished" podID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerID="a69fbc269e8d944e23b5a69faa9d6c995295f10f2a52d47e78b8bbbcd4937384" exitCode=60 Nov 03 17:00:21.747028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.746956 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerDied","Data":"a69fbc269e8d944e23b5a69faa9d6c995295f10f2a52d47e78b8bbbcd4937384"} Nov 03 17:00:21.747384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.747365 6001 scope.go:122] "RemoveContainer" containerID="a69fbc269e8d944e23b5a69faa9d6c995295f10f2a52d47e78b8bbbcd4937384" Nov 03 17:00:21.747706 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:21.747680 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw_odh-ai-gateway-infra(2b3d01a4-8458-4bbf-a98e-735627bfad04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" Nov 03 17:00:21.755878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.755849 6001 scope.go:122] "RemoveContainer" containerID="1c20d16779fdb852e3f0dce48770d9470db52e9c75f92d0eef6b8eb1b306b9a5" Nov 03 17:00:21.975997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:21.975196 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:22.182601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.180486 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\" (UniqueName: \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\") pod \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " Nov 03 17:00:22.182601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.180662 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\") pod \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\" (UID: \"86fc9ebc-6f86-4853-84ab-b09f38817d5c\") " Nov 03 17:00:22.200317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.200274 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca" pod "86fc9ebc-6f86-4853-84ab-b09f38817d5c" (UID: "86fc9ebc-6f86-4853-84ab-b09f38817d5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:22.210080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.209901 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:22.231059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.231024 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86fc9ebc-6f86-4853-84ab-b09f38817d5c-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.258220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.257830 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:22.276262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.272610 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:22.278071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.278040 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895420-lw7hq"] Nov 03 17:00:22.287898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.287846 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv" pod "86fc9ebc-6f86-4853-84ab-b09f38817d5c" (UID: "86fc9ebc-6f86-4853-84ab-b09f38817d5c"). InnerVolumeSpecName "kube-api-access-kb6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:22.370144 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:22.370088 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:00:22.374607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.374189 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\") pod \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " Nov 03 17:00:22.374982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.374957 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\") pod \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " Nov 03 17:00:22.375665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.375498 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca" pod "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" (UID: "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:22.375938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.375910 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\" (UniqueName: \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\") pod \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\" (UID: \"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1\") " Nov 03 17:00:22.377409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.377381 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.377905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.377888 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb6lv\" (UniqueName: \"kubernetes.io/projected/86fc9ebc-6f86-4853-84ab-b09f38817d5c-kube-api-access-kb6lv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.378035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.377843 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca" pod "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" (UID: "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:22.380709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.380674 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6" pod "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" (UID: "8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1"). InnerVolumeSpecName "kube-api-access-xzvg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:22.479936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.479894 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzvg6\" (UniqueName: \"kubernetes.io/projected/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-kube-api-access-xzvg6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.479936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.479931 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.824567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.824532 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" Nov 03 17:00:22.833322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.833288 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f558566-002f-4029-bd8f-d84e571b57eb" path="/var/lib/kubelet/pods/0f558566-002f-4029-bd8f-d84e571b57eb/volumes" Nov 03 17:00:22.833509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.833392 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" Nov 03 17:00:22.834606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.834587 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" path="/var/lib/kubelet/pods/1c51ae5a-63de-45e4-ad09-fecfba01f6fa/volumes" Nov 03 17:00:22.835156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.835135 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" path="/var/lib/kubelet/pods/2acda7e9-8a14-4301-a64e-ccd1accd8997/volumes" Nov 03 17:00:22.836100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836074 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k" event={"ID":"8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1","Type":"ContainerDied","Data":"32f438e517ef2a0b96314f452244ec2b6a7017838691081d5f0cd0b1692d2138"} Nov 03 17:00:22.836190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836126 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f438e517ef2a0b96314f452244ec2b6a7017838691081d5f0cd0b1692d2138" Nov 03 17:00:22.836190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836143 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" event={"ID":"2ed72ab3-ee6a-43f9-a671-4206ef791512","Type":"ContainerStarted","Data":"d07d750b1dfee2aa81e823a4b11a3de35d35d1ca6eb8291986300e0db512bf1f"} Nov 03 17:00:22.836190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836170 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerStarted","Data":"015c35bb4129ad99cf832b380b72f461b98cd7546f8c246d24a9e5fda24a771d"} Nov 03 17:00:22.836312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836194 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs" event={"ID":"86fc9ebc-6f86-4853-84ab-b09f38817d5c","Type":"ContainerDied","Data":"7be8aab2f301932789fbd693fbf5d8b7b4856253372e98095f3097daeef4018c"} Nov 03 17:00:22.836312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:22.836227 6001 scope.go:122] "RemoveContainer" containerID="13908deea35ad123875daf7d4994dd2dd821c4a0c378ebcee51d005ec1b7a8e1" Nov 03 17:00:23.023070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.023005 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" podStartSLOduration=3.022985152 podStartE2EDuration="3.022985152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:23.019083505 +0000 UTC m=+2718.352297074" watchObservedRunningTime="2026-11-03 17:00:23.022985152 +0000 UTC m=+2718.356198722" Nov 03 17:00:23.076623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.076584 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:23.080372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.080342 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895420-8nfqs"] Nov 03 17:00:23.097295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.097271 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:23.113040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.112988 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\") pod \"2b3d01a4-8458-4bbf-a98e-735627bfad04\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " Nov 03 17:00:23.113571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.113131 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\" (UniqueName: \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\") pod \"2b3d01a4-8458-4bbf-a98e-735627bfad04\" (UID: \"2b3d01a4-8458-4bbf-a98e-735627bfad04\") " Nov 03 17:00:23.115669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.115628 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca" pod "2b3d01a4-8458-4bbf-a98e-735627bfad04" (UID: "2b3d01a4-8458-4bbf-a98e-735627bfad04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:23.121499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.121396 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq" pod "2b3d01a4-8458-4bbf-a98e-735627bfad04" (UID: "2b3d01a4-8458-4bbf-a98e-735627bfad04"). InnerVolumeSpecName "kube-api-access-qj7gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:23.215940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.215905 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7gq\" (UniqueName: \"kubernetes.io/projected/2b3d01a4-8458-4bbf-a98e-735627bfad04-kube-api-access-qj7gq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:23.215940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.215937 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3d01a4-8458-4bbf-a98e-735627bfad04-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:23.722735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.722698 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:00:23.723843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723819 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.723843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723842 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723873 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723881 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723892 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723901 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723914 6001 cpu_manager.go:414] "containerMap: removing container" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723921 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723935 6001 cpu_manager.go:414] "containerMap: removing container" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723942 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723965 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723972 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723984 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" containerName="revoke-keys" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.723991 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" containerName="revoke-keys" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724018 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724025 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724035 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724044 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724229 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acda7e9-8a14-4301-a64e-ccd1accd8997" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724240 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724251 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724260 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724271 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724282 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724292 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d017ee1-1f5e-4ebc-814d-72db07b0f010" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724300 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" containerName="revoke-keys" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724309 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e08b8db-cd60-4eb7-a590-3f3f0c6104ba" containerName="cleanup" Nov 03 17:00:23.724608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.724322 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:23.728684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.728663 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:23.757465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.752299 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:00:23.953774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.953736 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-da409b\"" Nov 03 17:00:23.960483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.954119 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-da409b-dockercfg-v7g4m\"" Nov 03 17:00:23.978483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.975656 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:00:23.978483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.975717 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 17:00:23.978483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.976005 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:00:24.000466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:23.979927 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb4wt\" (UniqueName: \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.052661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.052626 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.087895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.083243 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" event={"ID":"2b3d01a4-8458-4bbf-a98e-735627bfad04","Type":"ContainerDied","Data":"61240c16b1fbf0497a5dcd66182be14cc3d35ca74f4b8ba9546f8be01d6c9b59"} Nov 03 17:00:24.088238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.088215 6001 scope.go:122] "RemoveContainer" containerID="a69fbc269e8d944e23b5a69faa9d6c995295f10f2a52d47e78b8bbbcd4937384" Nov 03 17:00:24.088436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.084569 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw" Nov 03 17:00:24.110337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.110310 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ed72ab3-ee6a-43f9-a671-4206ef791512" containerID="d07d750b1dfee2aa81e823a4b11a3de35d35d1ca6eb8291986300e0db512bf1f" exitCode=0 Nov 03 17:00:24.111385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.111361 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" event={"ID":"2ed72ab3-ee6a-43f9-a671-4206ef791512","Type":"ContainerDied","Data":"d07d750b1dfee2aa81e823a4b11a3de35d35d1ca6eb8291986300e0db512bf1f"} Nov 03 17:00:24.171274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.160116 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.171274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.170224 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb4wt\" (UniqueName: \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.172025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.171999 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.194990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.194959 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb4wt\" (UniqueName: \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\") pod \"payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.202476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.202435 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:24.206023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.205998 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895420-2kdcw"] Nov 03 17:00:24.368818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.368728 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:24.689832 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:24.689798 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef783c38_2ccf_4c46_80e0_8c7246752f15.slice/crio-140f2f0529d0458236d8820bd9ed641e21bada06718e0703cac26df2cf78672e WatchSource:0}: Error finding container 140f2f0529d0458236d8820bd9ed641e21bada06718e0703cac26df2cf78672e: Status 404 returned error can't find the container with id 140f2f0529d0458236d8820bd9ed641e21bada06718e0703cac26df2cf78672e Nov 03 17:00:24.690563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.690537 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:00:24.842204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.841739 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:00:24.842204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.841797 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 17:00:24.873464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.868033 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" path="/var/lib/kubelet/pods/2b3d01a4-8458-4bbf-a98e-735627bfad04/volumes" Nov 03 17:00:24.873464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:24.868777 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" path="/var/lib/kubelet/pods/86fc9ebc-6f86-4853-84ab-b09f38817d5c/volumes" Nov 03 17:00:25.120425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.120379 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" event={"ID":"ef783c38-2ccf-4c46-80e0-8c7246752f15","Type":"ContainerStarted","Data":"a4352befcab86d67f6b5e0f1642177d563a9152d161bf3258ca5efb64912446a"} Nov 03 17:00:25.120633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.120463 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" event={"ID":"ef783c38-2ccf-4c46-80e0-8c7246752f15","Type":"ContainerStarted","Data":"140f2f0529d0458236d8820bd9ed641e21bada06718e0703cac26df2cf78672e"} Nov 03 17:00:25.120633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.120495 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:25.237936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.237881 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" podStartSLOduration=2.237866152 podStartE2EDuration="2.237866152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:25.166643377 +0000 UTC m=+2720.499856947" watchObservedRunningTime="2026-11-03 17:00:25.237866152 +0000 UTC m=+2720.571079782" Nov 03 17:00:25.238401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.238373 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 17:00:25.239092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.239063 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" containerID="cri-o://b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4" gracePeriod=30 Nov 03 17:00:25.496035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.496006 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:25.577484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.557139 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\") pod \"2ed72ab3-ee6a-43f9-a671-4206ef791512\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " Nov 03 17:00:25.577484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.557276 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\" (UniqueName: \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\") pod \"2ed72ab3-ee6a-43f9-a671-4206ef791512\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " Nov 03 17:00:25.581321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.580591 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\") pod \"2ed72ab3-ee6a-43f9-a671-4206ef791512\" (UID: \"2ed72ab3-ee6a-43f9-a671-4206ef791512\") " Nov 03 17:00:25.581671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.581645 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca" pod "2ed72ab3-ee6a-43f9-a671-4206ef791512" (UID: "2ed72ab3-ee6a-43f9-a671-4206ef791512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:25.586908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.586881 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca" pod "2ed72ab3-ee6a-43f9-a671-4206ef791512" (UID: "2ed72ab3-ee6a-43f9-a671-4206ef791512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:25.592575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.592520 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh" pod "2ed72ab3-ee6a-43f9-a671-4206ef791512" (UID: "2ed72ab3-ee6a-43f9-a671-4206ef791512"). InnerVolumeSpecName "kube-api-access-2cmhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.628453 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.628381 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 17:00:25.683375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.683299 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.683375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.683330 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed72ab3-ee6a-43f9-a671-4206ef791512-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.683375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.683347 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cmhh\" (UniqueName: \"kubernetes.io/projected/2ed72ab3-ee6a-43f9-a671-4206ef791512-kube-api-access-2cmhh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.730091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.730061 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 17:00:25.783948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.783916 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784008 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784071 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784119 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784178 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784217 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784277 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784348 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784515 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.784640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784563 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\") pod \"da584ac1-244e-4ce8-bf4e-819777f46f6d\" (UID: \"da584ac1-244e-4ce8-bf4e-819777f46f6d\") " Nov 03 17:00:25.784940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.784916 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.785147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.785121 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.785224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.785190 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.785224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.785214 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.785344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.785267 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:25.785774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.785755 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.787772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.787733 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.789064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.789019 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "kube-api-access-8qp4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.789633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.789597 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.790413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.790388 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo" pod "da584ac1-244e-4ce8-bf4e-819777f46f6d" (UID: "da584ac1-244e-4ce8-bf4e-819777f46f6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:25.794521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.794501 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 17:00:25.836464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.832818 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:00:25.836464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.832898 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886380 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886428 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886508 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886553 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886590 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886632 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886695 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886786 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886807 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886866 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886905 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.886949 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887016 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887044 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887163 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887198 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\") pod \"95e48b6d-a1ab-43e0-888a-8549109703fc\" (UID: \"95e48b6d-a1ab-43e0-888a-8549109703fc\") " Nov 03 17:00:25.887251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.887266 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\") pod \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\" (UID: \"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078\") " Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888375 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888395 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qp4k\" (UniqueName: \"kubernetes.io/projected/da584ac1-244e-4ce8-bf4e-819777f46f6d-kube-api-access-8qp4k\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888405 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888418 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888428 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888437 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da584ac1-244e-4ce8-bf4e-819777f46f6d-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.888688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.888479 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da584ac1-244e-4ce8-bf4e-819777f46f6d-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.890985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.889609 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:25.891670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.891637 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.891910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.891883 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.892298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.892275 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.893772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.892860 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:25.893772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.892953 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.893772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.893124 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.893772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.893544 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.893772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.893738 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.894106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.893936 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.894163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.894137 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:25.895783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.895755 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.895905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.895866 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "kube-api-access-xthnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.899610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.897241 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.899610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.897531 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.899610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.899577 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "kube-api-access-tvjbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:25.900293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.900267 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy" pod "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" (UID: "2ce4183a-b3a2-4f7f-a32d-4a59a37b6078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:25.902390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.902371 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo" pod "95e48b6d-a1ab-43e0-888a-8549109703fc" (UID: "95e48b6d-a1ab-43e0-888a-8549109703fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:25.991675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991642 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991673 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991688 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95e48b6d-a1ab-43e0-888a-8549109703fc-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991698 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjbn\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-kube-api-access-tvjbn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991708 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991718 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991726 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991734 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xthnx\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-kube-api-access-xthnx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991742 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991751 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991759 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991766 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991774 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991781 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991788 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95e48b6d-a1ab-43e0-888a-8549109703fc-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991795 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95e48b6d-a1ab-43e0-888a-8549109703fc-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991802 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:25.991867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:25.991811 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:26.129850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.129811 6001 generic.go:356] "Generic (PLEG): container finished" podID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerID="02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec" exitCode=0 Nov 03 17:00:26.130029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.129875 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" Nov 03 17:00:26.130029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.129885 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" event={"ID":"95e48b6d-a1ab-43e0-888a-8549109703fc","Type":"ContainerDied","Data":"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec"} Nov 03 17:00:26.130029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.129967 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp" event={"ID":"95e48b6d-a1ab-43e0-888a-8549109703fc","Type":"ContainerDied","Data":"06aed220ab853848ea5f9366bc1fa76bade6c64607f2157e16b1e306b6b5d9ab"} Nov 03 17:00:26.130029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.129991 6001 scope.go:122] "RemoveContainer" containerID="02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec" Nov 03 17:00:26.131756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.131623 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerID="58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca" exitCode=0 Nov 03 17:00:26.131756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.131660 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" event={"ID":"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078","Type":"ContainerDied","Data":"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca"} Nov 03 17:00:26.131756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.131692 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" Nov 03 17:00:26.131756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.131714 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk" event={"ID":"2ce4183a-b3a2-4f7f-a32d-4a59a37b6078","Type":"ContainerDied","Data":"bfc09420b6dfdbfd7c29a832d0e6cb497508b78d76fab1ca700543c786237fd0"} Nov 03 17:00:26.139750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.139390 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" Nov 03 17:00:26.139750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.139537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd" event={"ID":"2ed72ab3-ee6a-43f9-a671-4206ef791512","Type":"ContainerDied","Data":"3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa"} Nov 03 17:00:26.139750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.139579 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf20875f7849ad3011b49ca0cf04127afd14cf21c96e09440a2e5667fd48cfa" Nov 03 17:00:26.142942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.142921 6001 generic.go:356] "Generic (PLEG): container finished" podID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerID="09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1" exitCode=0 Nov 03 17:00:26.143091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.142970 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" Nov 03 17:00:26.143091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.143006 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" event={"ID":"da584ac1-244e-4ce8-bf4e-819777f46f6d","Type":"ContainerDied","Data":"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1"} Nov 03 17:00:26.143091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.143053 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg" event={"ID":"da584ac1-244e-4ce8-bf4e-819777f46f6d","Type":"ContainerDied","Data":"7ead556c62d0a241376e1feefe0ccd2ec8de8e5ea9ef743fa7ee1b88d357483c"} Nov 03 17:00:26.169216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.169184 6001 scope.go:122] "RemoveContainer" containerID="02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec" Nov 03 17:00:26.169770 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:26.169736 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec\": container with ID starting with 02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec not found: ID does not exist" containerID="02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec" Nov 03 17:00:26.169894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.169779 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec"} err="failed to get container status \"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec\": rpc error: code = NotFound desc = could not find container \"02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec\": container with ID starting with 02b820f80d8fc1100645c20d0c106b322ca5f3872aae69fef45cc5601b4589ec not found: ID does not exist" Nov 03 17:00:26.169894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.169802 6001 scope.go:122] "RemoveContainer" containerID="58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca" Nov 03 17:00:26.186203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.186183 6001 scope.go:122] "RemoveContainer" containerID="58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca" Nov 03 17:00:26.186591 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:26.186561 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca\": container with ID starting with 58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca not found: ID does not exist" containerID="58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca" Nov 03 17:00:26.186705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.186601 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca"} err="failed to get container status \"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca\": rpc error: code = NotFound desc = could not find container \"58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca\": container with ID starting with 58e71bf22e47cbec6dc34333f88b7400933174a19d71160445fb9a65e35ce7ca not found: ID does not exist" Nov 03 17:00:26.186705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.186630 6001 scope.go:122] "RemoveContainer" containerID="09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1" Nov 03 17:00:26.196539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.196521 6001 scope.go:122] "RemoveContainer" containerID="09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1" Nov 03 17:00:26.196802 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:26.196783 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1\": container with ID starting with 09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1 not found: ID does not exist" containerID="09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1" Nov 03 17:00:26.196869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.196809 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1"} err="failed to get container status \"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1\": rpc error: code = NotFound desc = could not find container \"09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1\": container with ID starting with 09bf24f38090eb966d68f64798a206f0d84e0d40f8155b1595993d4eae7ceed1 not found: ID does not exist" Nov 03 17:00:26.228084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.228059 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 17:00:26.278097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.277875 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vkfvk"] Nov 03 17:00:26.322738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.322708 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 17:00:26.329067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.329042 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-vcvfp"] Nov 03 17:00:26.394998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.394905 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 17:00:26.398635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.398600 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kczcg"] Nov 03 17:00:26.519635 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:26.519598 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:26.792085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.792051 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" path="/var/lib/kubelet/pods/2ce4183a-b3a2-4f7f-a32d-4a59a37b6078/volumes" Nov 03 17:00:26.792670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.792653 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" path="/var/lib/kubelet/pods/95e48b6d-a1ab-43e0-888a-8549109703fc/volumes" Nov 03 17:00:26.793169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.793154 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" path="/var/lib/kubelet/pods/da584ac1-244e-4ce8-bf4e-819777f46f6d/volumes" Nov 03 17:00:26.831919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.831887 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:00:26.832066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:26.831950 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 17:00:27.910334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:27.910301 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:00:27.926492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:27.917593 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.241:15021/healthz/ready\": dial tcp 10.134.0.241:15021: connect: connection refused" Nov 03 17:00:28.836231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:28.836203 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:00:28.836722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:28.836709 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:00:28.837571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:28.837550 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:00:30.474045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.474019 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 17:00:30.625621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625539 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.625778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625622 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.625778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625646 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.625778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625689 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.625778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625718 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.625778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625765 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.626051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625803 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:30.626051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625849 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.626051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625916 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.626051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.625940 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:30.626051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626009 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\") pod \"07a9efe6-aeb6-4161-8ec3-180c5b963891\" (UID: \"07a9efe6-aeb6-4161-8ec3-180c5b963891\") " Nov 03 17:00:30.626325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626164 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:30.626325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626215 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:30.626595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626570 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:30.626822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626801 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07a9efe6-aeb6-4161-8ec3-180c5b963891-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.626893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626828 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.626893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626846 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.626893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626860 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.626893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.626874 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.628031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.628003 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:30.628134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.628051 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "kube-api-access-v2bt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:30.628214 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.628195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:30.628270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.628229 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy" pod "07a9efe6-aeb6-4161-8ec3-180c5b963891" (UID: "07a9efe6-aeb6-4161-8ec3-180c5b963891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:30.729213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.729176 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2bt9\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-kube-api-access-v2bt9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.729213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.729208 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.729213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.729218 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.729558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:30.729225 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07a9efe6-aeb6-4161-8ec3-180c5b963891-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:31.217764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.217729 6001 generic.go:356] "Generic (PLEG): container finished" podID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerID="b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4" exitCode=0 Nov 03 17:00:31.217949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.217847 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" Nov 03 17:00:31.217949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.217877 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" event={"ID":"07a9efe6-aeb6-4161-8ec3-180c5b963891","Type":"ContainerDied","Data":"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4"} Nov 03 17:00:31.217949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.217935 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq" event={"ID":"07a9efe6-aeb6-4161-8ec3-180c5b963891","Type":"ContainerDied","Data":"8b7e16779d7f36889b421b3e971563329fecc7764486f075b91c8fc74e30cd1b"} Nov 03 17:00:31.218091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.217965 6001 scope.go:122] "RemoveContainer" containerID="b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4" Nov 03 17:00:31.230108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.230082 6001 scope.go:122] "RemoveContainer" containerID="b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4" Nov 03 17:00:31.230428 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:31.230406 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4\": container with ID starting with b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4 not found: ID does not exist" containerID="b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4" Nov 03 17:00:31.230428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.230435 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4"} err="failed to get container status \"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4\": rpc error: code = NotFound desc = could not find container \"b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4\": container with ID starting with b2f3365a3a82f7e99613e31880c2a6495167a8475d28c2db67a60d80d79f16f4 not found: ID does not exist" Nov 03 17:00:31.275880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.275849 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 17:00:31.288752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:31.288726 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-z52lq"] Nov 03 17:00:32.791597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:32.791567 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" path="/var/lib/kubelet/pods/07a9efe6-aeb6-4161-8ec3-180c5b963891/volumes" Nov 03 17:00:36.149677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:36.149648 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:00:36.538214 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:36.538173 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:36.655370 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:36.655331 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:39.485606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.485571 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:00:39.486644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486613 6001 cpu_manager.go:414] "containerMap: removing container" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486644 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486665 6001 cpu_manager.go:414] "containerMap: removing container" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486673 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486687 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ed72ab3-ee6a-43f9-a671-4206ef791512" containerName="revoke-keys" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486694 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed72ab3-ee6a-43f9-a671-4206ef791512" containerName="revoke-keys" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486716 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" Nov 03 17:00:39.486751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486724 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486760 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486766 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486780 6001 cpu_manager.go:414] "containerMap: removing container" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486786 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486800 6001 cpu_manager.go:414] "containerMap: removing container" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486806 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" Nov 03 17:00:39.486980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486978 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="da584ac1-244e-4ce8-bf4e-819777f46f6d" containerName="istio-proxy" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.486992 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a9efe6-aeb6-4161-8ec3-180c5b963891" containerName="istio-proxy" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487003 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed72ab3-ee6a-43f9-a671-4206ef791512" containerName="revoke-keys" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487015 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce4183a-b3a2-4f7f-a32d-4a59a37b6078" containerName="istio-proxy" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487028 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fc9ebc-6f86-4853-84ab-b09f38817d5c" containerName="cleanup" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487038 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e48b6d-a1ab-43e0-888a-8549109703fc" containerName="istio-proxy" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487046 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b3d01a4-8458-4bbf-a98e-735627bfad04" containerName="cleanup" Nov 03 17:00:39.487240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.487060 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c51ae5a-63de-45e4-ad09-fecfba01f6fa" containerName="cleanup" Nov 03 17:00:39.490643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.490616 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512461 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512634 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512661 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512799 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512816 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.512946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.512852 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.513047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.513019 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.513147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.513121 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8vf\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.513257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.513165 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.513935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.513915 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:00:39.523233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.523204 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d6fcae-openshift-default-dockercfg-7qz7l\"" Nov 03 17:00:39.614439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614404 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614478 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614537 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614554 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614592 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614664 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614709 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8vf\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614842 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614867 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.614977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614959 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.615132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.614961 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.615132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.615058 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.615271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.615253 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.615330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.615314 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.617196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.617174 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.617367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.617346 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.647797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.647720 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.650927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.650867 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8vf\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:39.807564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:39.807222 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:40.143689 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:40.143659 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ed6a75_097f_471c_92f9_2356c11edb19.slice/crio-a50d2f06730828c014b349f6ea8d8b8cc1ad19f631002f751939e8f737f9260d WatchSource:0}: Error finding container a50d2f06730828c014b349f6ea8d8b8cc1ad19f631002f751939e8f737f9260d: Status 404 returned error can't find the container with id a50d2f06730828c014b349f6ea8d8b8cc1ad19f631002f751939e8f737f9260d Nov 03 17:00:40.146325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.146300 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:00:40.147140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.147106 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:40.147241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.147181 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:40.147308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.147250 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:00:40.335044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.335009 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerStarted","Data":"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385"} Nov 03 17:00:40.335213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.335069 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerStarted","Data":"a50d2f06730828c014b349f6ea8d8b8cc1ad19f631002f751939e8f737f9260d"} Nov 03 17:00:40.399230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.399130 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podStartSLOduration=1.39911515 podStartE2EDuration="1.39911515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:40.396328384 +0000 UTC m=+2735.729541979" watchObservedRunningTime="2026-11-03 17:00:40.39911515 +0000 UTC m=+2735.732328718" Nov 03 17:00:40.808047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:40.808016 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:41.835505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:41.835155 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:42.354959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:42.354925 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:42.359629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:42.359600 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:00:44.876904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.876870 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:00:44.881545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.881529 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:44.906735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.906709 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d6fcae\"" Nov 03 17:00:44.908908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.908877 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:00:44.919688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.919667 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d6fcae-dockercfg-zp2l2\"" Nov 03 17:00:44.933894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.933864 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65nfg\" (UniqueName: \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:44.934003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.933936 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:44.945533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.945503 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:00:44.951108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.951090 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:44.969003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:44.968982 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:00:45.060341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.058053 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65nfg\" (UniqueName: \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:45.060341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.058134 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:45.060341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.059233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:45.164186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.163214 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.164186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.163473 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.205961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.203555 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65nfg\" (UniqueName: \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\") pod \"payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:45.264894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.264859 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.265119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.264991 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.265950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.265926 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.290962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.290932 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\") pod \"payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.497429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.497400 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:45.574248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.574209 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:45.806806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.806762 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:00:45.903280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:45.903252 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:00:45.904935 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:00:45.904910 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2617b18_379e_402d_9021_42c78c68d4be.slice/crio-21c19d1853346b41a6121fd70eca224f574a8636f524c940eb30acc0a5a82feb WatchSource:0}: Error finding container 21c19d1853346b41a6121fd70eca224f574a8636f524c940eb30acc0a5a82feb: Status 404 returned error can't find the container with id 21c19d1853346b41a6121fd70eca224f574a8636f524c940eb30acc0a5a82feb Nov 03 17:00:46.391928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.391889 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" event={"ID":"b2617b18-379e-402d-9021-42c78c68d4be","Type":"ContainerStarted","Data":"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a"} Nov 03 17:00:46.392149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.391951 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" event={"ID":"b2617b18-379e-402d-9021-42c78c68d4be","Type":"ContainerStarted","Data":"21c19d1853346b41a6121fd70eca224f574a8636f524c940eb30acc0a5a82feb"} Nov 03 17:00:46.392149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.391999 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:46.393728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.393700 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" event={"ID":"d52f73af-1985-484f-9c57-a4683e472589","Type":"ContainerStarted","Data":"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0"} Nov 03 17:00:46.393852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.393740 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" event={"ID":"d52f73af-1985-484f-9c57-a4683e472589","Type":"ContainerStarted","Data":"5a8f0954f8a8be5322efea64fdf2f1d5ed2242138edadc296ebbed696df2da8e"} Nov 03 17:00:46.393852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.393782 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:46.439670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.439618 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" podStartSLOduration=2.43960387 podStartE2EDuration="2.43960387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:46.436948382 +0000 UTC m=+2741.770161962" watchObservedRunningTime="2026-11-03 17:00:46.43960387 +0000 UTC m=+2741.772817462" Nov 03 17:00:46.483570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:46.483522 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" podStartSLOduration=2.48350791 podStartE2EDuration="2.48350791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:46.480136079 +0000 UTC m=+2741.813349660" watchObservedRunningTime="2026-11-03 17:00:46.48350791 +0000 UTC m=+2741.816721479" Nov 03 17:00:46.668170 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:46.668093 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:48.826306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:48.826269 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 17:00:48.826896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:48.826543 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" containerID="cri-o://35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117" gracePeriod=30 Nov 03 17:00:51.076258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:51.076224 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 17:00:51.076739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:51.076461 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" containerID="cri-o://3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe" gracePeriod=30 Nov 03 17:00:51.729844 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:51.729787 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:52.253055 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:52.253017 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:00:52.931583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:52.931428 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:00:52.931583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:52.931557 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:00:54.110226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.110166 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 17:00:54.222863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.222829 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.222885 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.222975 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223020 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223158 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223161 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:54.223256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223202 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223257 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223289 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223381 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\") pod \"db03d546-7b51-4c71-be04-8791f7985d18\" (UID: \"db03d546-7b51-4c71-be04-8791f7985d18\") " Nov 03 17:00:54.223627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223503 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:54.223708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223690 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:54.223937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.223909 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:54.224418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.224253 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.224418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.224283 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.224418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.224297 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.224418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.224311 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.224917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.224802 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:54.231982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.231950 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:54.232143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.232126 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:54.232544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.232521 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:54.234840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.234816 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7" pod "db03d546-7b51-4c71-be04-8791f7985d18" (UID: "db03d546-7b51-4c71-be04-8791f7985d18"). InnerVolumeSpecName "kube-api-access-6hgh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:54.325515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.325474 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgh7\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-kube-api-access-6hgh7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.325515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.325509 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db03d546-7b51-4c71-be04-8791f7985d18-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.325515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.325519 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db03d546-7b51-4c71-be04-8791f7985d18-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.325515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.325527 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db03d546-7b51-4c71-be04-8791f7985d18-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.325817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.325534 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db03d546-7b51-4c71-be04-8791f7985d18-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:54.474092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.474054 6001 generic.go:356] "Generic (PLEG): container finished" podID="db03d546-7b51-4c71-be04-8791f7985d18" containerID="35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117" exitCode=0 Nov 03 17:00:54.474289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.474111 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" Nov 03 17:00:54.474289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.474136 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerDied","Data":"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117"} Nov 03 17:00:54.474289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.474192 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8" event={"ID":"db03d546-7b51-4c71-be04-8791f7985d18","Type":"ContainerDied","Data":"2271af374cd1e3e4a9db9c77506cc9880a2454d043e51a033c2eec4ea0d00c6f"} Nov 03 17:00:54.474289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.474205 6001 scope.go:122] "RemoveContainer" containerID="35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117" Nov 03 17:00:54.487408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.487336 6001 scope.go:122] "RemoveContainer" containerID="8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31" Nov 03 17:00:54.497467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.497433 6001 scope.go:122] "RemoveContainer" containerID="35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117" Nov 03 17:00:54.497719 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:54.497700 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117\": container with ID starting with 35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117 not found: ID does not exist" containerID="35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117" Nov 03 17:00:54.497783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.497726 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117"} err="failed to get container status \"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117\": rpc error: code = NotFound desc = could not find container \"35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117\": container with ID starting with 35ccf3b7b6502199ff47e03a3995e714b83f6c9cfe14e85b4ad1f557388a1117 not found: ID does not exist" Nov 03 17:00:54.497783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.497744 6001 scope.go:122] "RemoveContainer" containerID="8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31" Nov 03 17:00:54.497998 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:54.497980 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31\": container with ID starting with 8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31 not found: ID does not exist" containerID="8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31" Nov 03 17:00:54.498046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.498003 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31"} err="failed to get container status \"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31\": rpc error: code = NotFound desc = could not find container \"8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31\": container with ID starting with 8406b39725ac0d878986e441256a0d198ea0425ad116dc9fbfa730f42779db31 not found: ID does not exist" Nov 03 17:00:54.531709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.531688 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 17:00:54.543225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.543195 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-v74r8"] Nov 03 17:00:54.791175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:54.791105 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db03d546-7b51-4c71-be04-8791f7985d18" path="/var/lib/kubelet/pods/db03d546-7b51-4c71-be04-8791f7985d18/volumes" Nov 03 17:00:56.382627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.382591 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 17:00:56.496335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.496303 6001 generic.go:356] "Generic (PLEG): container finished" podID="76d011fb-b13f-447e-962a-4601bb81eadb" containerID="3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe" exitCode=0 Nov 03 17:00:56.496531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.496350 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" Nov 03 17:00:56.496531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.496406 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" event={"ID":"76d011fb-b13f-447e-962a-4601bb81eadb","Type":"ContainerDied","Data":"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe"} Nov 03 17:00:56.496531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.496487 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs" event={"ID":"76d011fb-b13f-447e-962a-4601bb81eadb","Type":"ContainerDied","Data":"170220a44b8223b825d96dec1c139fd7e2d6b596f03daa19cfae72acbd397785"} Nov 03 17:00:56.496531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.496525 6001 scope.go:122] "RemoveContainer" containerID="3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe" Nov 03 17:00:56.507806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.507780 6001 scope.go:122] "RemoveContainer" containerID="3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe" Nov 03 17:00:56.508092 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:56.508069 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe\": container with ID starting with 3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe not found: ID does not exist" containerID="3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe" Nov 03 17:00:56.508163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.508100 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe"} err="failed to get container status \"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe\": rpc error: code = NotFound desc = could not find container \"3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe\": container with ID starting with 3c0b4f431531398e935041600d89734c179d5efca7518c3e12d37435486629fe not found: ID does not exist" Nov 03 17:00:56.572127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572090 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572174 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572250 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572296 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572487 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572526 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572581 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572705 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.572835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.572797 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\") pod \"76d011fb-b13f-447e-962a-4601bb81eadb\" (UID: \"76d011fb-b13f-447e-962a-4601bb81eadb\") " Nov 03 17:00:56.574009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.573971 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:56.574756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.574730 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:56.575236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.575211 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:56.576128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.576107 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:56.576849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.576825 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:56.577141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.577122 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76d011fb-b13f-447e-962a-4601bb81eadb-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.577263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.577251 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.577358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.577346 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.577521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.577435 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.577716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.577699 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.580489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.580434 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:56.580798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.580700 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:56.580897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.580755 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:56.581783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.581757 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4" pod "76d011fb-b13f-447e-962a-4601bb81eadb" (UID: "76d011fb-b13f-447e-962a-4601bb81eadb"). InnerVolumeSpecName "kube-api-access-9tjm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:56.679509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.679406 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76d011fb-b13f-447e-962a-4601bb81eadb-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.679509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.679460 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.679509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.679476 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tjm4\" (UniqueName: \"kubernetes.io/projected/76d011fb-b13f-447e-962a-4601bb81eadb-kube-api-access-9tjm4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.679509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.679490 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76d011fb-b13f-447e-962a-4601bb81eadb-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.737961 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:00:56.737923 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d1f813_1e5e_4745_9391_dd03d3562c78.slice/crio-a79792a81a0f1ce8e26a084e4f66e9381f9067d13ecd4438aadc1a2fd70ba32a.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:56.858506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.857578 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 17:00:56.862714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:56.862680 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-82srs"] Nov 03 17:00:57.413409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:57.413375 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:00:57.416473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:57.416334 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:00:58.790457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:00:58.790419 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" path="/var/lib/kubelet/pods/76d011fb-b13f-447e-962a-4601bb81eadb/volumes" Nov 03 17:01:10.415423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:10.415389 6001 scope.go:122] "RemoveContainer" containerID="de4d04ea7156754542e2645dde770fe27443f44f1388352468fd87dc5293860b" Nov 03 17:01:10.426826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:10.426804 6001 scope.go:122] "RemoveContainer" containerID="c560dd8058f069c0fa380781f8b0525bee4ffddcf121a9ba5025e5ac76e045ea" Nov 03 17:01:22.136727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.136691 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:01:22.137511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137492 6001 cpu_manager.go:414] "containerMap: removing container" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.137511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137510 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.137673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137553 6001 cpu_manager.go:414] "containerMap: removing container" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" Nov 03 17:01:22.137673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137558 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" Nov 03 17:01:22.137673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137672 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d011fb-b13f-447e-962a-4601bb81eadb" containerName="istio-proxy" Nov 03 17:01:22.137838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.137684 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.143651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.143629 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.167700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.167668 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:01:22.228497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.228467 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e2d835-openshift-default-dockercfg-8f5zz\"" Nov 03 17:01:22.236843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.236797 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqkh2\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237279 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237477 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237592 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237672 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237721 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237836 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.237965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237888 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.238026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.237979 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.270486 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:01:22.269701 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:01:22.340170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.340125 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqkh2\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.340933 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.340904 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.341628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.341504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.341916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.341896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.349301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.349237 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354009 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354111 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354185 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354315 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.357203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354345 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.367091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.363142 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.371696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.354421 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.372943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.372889 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.378266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.378231 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqkh2\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.379250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.379222 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.383212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.383158 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.396565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.396478 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.475803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.475764 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:22.735359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.735315 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:01:22.736542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.736516 6001 cpu_manager.go:414] "containerMap: removing container" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.736542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.736541 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.736764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.736750 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="db03d546-7b51-4c71-be04-8791f7985d18" containerName="istio-proxy" Nov 03 17:01:22.742510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.742484 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:22.759059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.758684 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6046c2\"" Nov 03 17:01:22.761487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.761405 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:01:22.761708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.761680 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6046c2-dockercfg-jh629\"" Nov 03 17:01:22.989234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.987044 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:22.989234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.987168 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzgv\" (UniqueName: \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:22.989234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.988099 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:01:22.991470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.991416 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:22.991685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.991596 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:22.991685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:22.991652 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:23.088610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.088573 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.088795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.088660 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzgv\" (UniqueName: \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.089518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.089491 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.113318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.113247 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzgv\" (UniqueName: \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\") pod \"payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.365324 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.365219 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.738548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.738521 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:01:23.742405 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:01:23.742376 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdecf6a8_9484_43ea_86da_5757e5aabaab.slice/crio-7040ba91445d809b760153e2828a526bdff6e2405018b8793dc619d6cf6ae99c WatchSource:0}: Error finding container 7040ba91445d809b760153e2828a526bdff6e2405018b8793dc619d6cf6ae99c: Status 404 returned error can't find the container with id 7040ba91445d809b760153e2828a526bdff6e2405018b8793dc619d6cf6ae99c Nov 03 17:01:23.964466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.956133 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" event={"ID":"fdecf6a8-9484-43ea-86da-5757e5aabaab","Type":"ContainerStarted","Data":"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22"} Nov 03 17:01:23.964466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.956205 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" event={"ID":"fdecf6a8-9484-43ea-86da-5757e5aabaab","Type":"ContainerStarted","Data":"7040ba91445d809b760153e2828a526bdff6e2405018b8793dc619d6cf6ae99c"} Nov 03 17:01:23.964466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.957433 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:23.969478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.968465 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" event={"ID":"1a8aa919-a43e-44b1-a9cc-d884d990a7ae","Type":"ContainerStarted","Data":"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42"} Nov 03 17:01:23.969478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:23.968532 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" event={"ID":"1a8aa919-a43e-44b1-a9cc-d884d990a7ae","Type":"ContainerStarted","Data":"8ac55c7cc6d8397fca7ae5e538ea6304f4af0cd11aaf88cd2a780b06ca769264"} Nov 03 17:01:24.009617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:24.009521 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" podStartSLOduration=2.009506648 podStartE2EDuration="2.009506648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:24.007787515 +0000 UTC m=+2779.341001083" watchObservedRunningTime="2026-11-03 17:01:24.009506648 +0000 UTC m=+2779.342720216" Nov 03 17:01:24.477649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:24.477613 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:24.479103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:24.479075 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:24.479239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:24.479129 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:25.476103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:25.476067 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.476331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:25.476119 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:26.476792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:26.476757 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.477190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:26.476812 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:27.502465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:27.499669 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.502465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:27.499734 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:28.476831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:28.476794 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.477031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:28.476850 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:29.478038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:29.476715 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body= Nov 03 17:01:29.478038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:29.477580 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" Nov 03 17:01:30.481471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:30.481422 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:30.481946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:30.481857 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:30.482714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:30.482693 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:01:30.564106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:30.564048 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podStartSLOduration=8.564034757 podStartE2EDuration="8.564034757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:24.070250187 +0000 UTC m=+2779.403463757" watchObservedRunningTime="2026-11-03 17:01:30.564034757 +0000 UTC m=+2785.897248325" Nov 03 17:01:35.060625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.060587 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:01:35.065745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.065705 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.086707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.086676 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3a0e27-openshift-default-dockercfg-s7k4t\"" Nov 03 17:01:35.088410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.088377 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:01:35.254709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.254670 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255006 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.254974 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zk8r\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255146 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255189 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255281 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255517 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255554 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255702 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.255855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.255806 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.362050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.361663 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zk8r\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.362050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.361837 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.362307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.361867 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.362307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362205 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362434 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362483 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362591 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362648 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.362739 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.363227 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.363365 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.363423 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.363692 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.371475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.364092 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.389962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.389926 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.457412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.456609 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.458172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.458146 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.504479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.480809 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zk8r\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.687264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.687159 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:35.991567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:35.991529 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:01:36.011694 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:01:36.011661 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3250e537_243b_4b0b_bcb1_fa5bba0f02a8.slice/crio-e364100db6ceb80971f0689183f0c3f90c851a416562bbc5cbb15ff4c1744e07 WatchSource:0}: Error finding container e364100db6ceb80971f0689183f0c3f90c851a416562bbc5cbb15ff4c1744e07: Status 404 returned error can't find the container with id e364100db6ceb80971f0689183f0c3f90c851a416562bbc5cbb15ff4c1744e07 Nov 03 17:01:36.014592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:36.014552 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:36.014702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:36.014639 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:36.014702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:36.014668 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:01:36.025005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:36.024976 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:01:36.077143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:36.077104 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerStarted","Data":"e364100db6ceb80971f0689183f0c3f90c851a416562bbc5cbb15ff4c1744e07"} Nov 03 17:01:37.089256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:37.089221 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerStarted","Data":"3a24a22b4f97b88583ad6a11c3d4606d9d43d342609e3c73ac2e9ba019d68fe0"} Nov 03 17:01:37.153327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:37.153262 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podStartSLOduration=2.153245797 podStartE2EDuration="2.153245797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:37.149484865 +0000 UTC m=+2792.482698435" watchObservedRunningTime="2026-11-03 17:01:37.153245797 +0000 UTC m=+2792.486459366" Nov 03 17:01:37.688396 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:37.688356 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:37.695001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:37.694972 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:38.097928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:38.097894 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:38.099058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:38.099029 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:01:43.444002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.443948 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:01:43.449201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.449172 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.469664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.469630 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:01:43.555481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.554262 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e0e63d\"" Nov 03 17:01:43.555481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.554530 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e0e63d-dockercfg-btz2c\"" Nov 03 17:01:43.668004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.667942 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:01:43.688569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.688509 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.688922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.688901 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtfd\" (UniqueName: \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.691179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.691150 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:01:43.691304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.691285 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.790406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.790372 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.790612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.790552 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtfd\" (UniqueName: \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.790700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.790687 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zh9\" (UniqueName: \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.790896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.790871 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.791576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.791559 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.815057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.815003 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtfd\" (UniqueName: \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\") pod \"payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:43.892249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.892196 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.892516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.892425 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zh9\" (UniqueName: \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.892993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.892974 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:43.917944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:43.917913 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zh9\" (UniqueName: \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\") pod \"payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:44.011911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:44.011875 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:44.068976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:44.068928 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:44.323458 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:01:44.323415 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2f9ab3_b8a9_4d82_8bcb_66a1a284b389.slice/crio-a615a5315f98a10dc59b4f1eaa43ff52c74f5252018457b075b1e6dd161b4a61 WatchSource:0}: Error finding container a615a5315f98a10dc59b4f1eaa43ff52c74f5252018457b075b1e6dd161b4a61: Status 404 returned error can't find the container with id a615a5315f98a10dc59b4f1eaa43ff52c74f5252018457b075b1e6dd161b4a61 Nov 03 17:01:44.324340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:44.324311 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:01:44.431059 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:01:44.431030 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7aa3bd7_b7fb_46e8_897e_1fd99f4f0842.slice/crio-fc279cf35cff4cd0358c8e784c5fdb4cc6a66d552081c35d8696e7619bff17df WatchSource:0}: Error finding container fc279cf35cff4cd0358c8e784c5fdb4cc6a66d552081c35d8696e7619bff17df: Status 404 returned error can't find the container with id fc279cf35cff4cd0358c8e784c5fdb4cc6a66d552081c35d8696e7619bff17df Nov 03 17:01:44.431743 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:44.431716 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:01:45.197152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.197114 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" event={"ID":"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842","Type":"ContainerStarted","Data":"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571"} Nov 03 17:01:45.197678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.197181 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" event={"ID":"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842","Type":"ContainerStarted","Data":"fc279cf35cff4cd0358c8e784c5fdb4cc6a66d552081c35d8696e7619bff17df"} Nov 03 17:01:45.197678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.197341 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:45.199104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.199070 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" event={"ID":"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389","Type":"ContainerStarted","Data":"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b"} Nov 03 17:01:45.199258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.199117 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" event={"ID":"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389","Type":"ContainerStarted","Data":"a615a5315f98a10dc59b4f1eaa43ff52c74f5252018457b075b1e6dd161b4a61"} Nov 03 17:01:45.199258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.199156 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:45.240558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.240502 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" podStartSLOduration=2.240487907 podStartE2EDuration="2.240487907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:45.239762282 +0000 UTC m=+2800.572975850" watchObservedRunningTime="2026-11-03 17:01:45.240487907 +0000 UTC m=+2800.573701476" Nov 03 17:01:45.284129 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:45.284072 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" podStartSLOduration=2.284057591 podStartE2EDuration="2.284057591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:45.28265398 +0000 UTC m=+2800.615867550" watchObservedRunningTime="2026-11-03 17:01:45.284057591 +0000 UTC m=+2800.617271160" Nov 03 17:01:52.276905 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:01:52.276802 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:01:56.210101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:56.210066 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:01:56.210704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:56.210159 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:01:58.395683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:58.395643 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:01:58.396142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:01:58.395712 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:02:00.371008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.370973 6001 generic.go:356] "Generic (PLEG): container finished" podID="77ed6a75-097f-471c-92f9-2356c11edb19" containerID="45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385" exitCode=0 Nov 03 17:02:00.371541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.371048 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerDied","Data":"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385"} Nov 03 17:02:00.371595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.371559 6001 scope.go:122] "RemoveContainer" containerID="45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385" Nov 03 17:02:00.373548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.373520 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:00.373642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.373589 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:00.373642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:00.373620 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:01.381437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.381389 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerStarted","Data":"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68"} Nov 03 17:02:01.615483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.615426 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:02:01.619337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.619309 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:01.639250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.639165 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ca9d7c\"" Nov 03 17:02:01.639250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.639187 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ca9d7c-dockercfg-2tv22\"" Nov 03 17:02:01.654470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.643528 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:02:01.759471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.737380 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4scv\" (UniqueName: \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:01.759950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.759916 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:01.770522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.744115 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:02:01.825047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.825006 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:02:01.825363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.825326 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:02:01.827261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.827237 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:02:01.827629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.827524 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:01.834028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.827600 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:02:01.867750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.863571 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4scv\" (UniqueName: \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:01.929471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.911942 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:01.949476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.933832 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:02.001381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:01.994288 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4scv\" (UniqueName: \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\") pod \"payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:02.040038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.040001 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.040222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.040116 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctd9d\" (UniqueName: \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.142207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.142172 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctd9d\" (UniqueName: \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.142455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.142426 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.143145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.143123 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.172409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.172315 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctd9d\" (UniqueName: \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\") pod \"payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.234496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.234462 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:02.262768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.262723 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:02.599115 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:02.599079 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda38d94fd_440f_41d7_9a8b_1a3b42fc378a.slice/crio-bfec078242c73e38d5fdc77453234034a349547c2cd43528c6b5e17b21f5499e WatchSource:0}: Error finding container bfec078242c73e38d5fdc77453234034a349547c2cd43528c6b5e17b21f5499e: Status 404 returned error can't find the container with id bfec078242c73e38d5fdc77453234034a349547c2cd43528c6b5e17b21f5499e Nov 03 17:02:02.600558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.600526 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:02:02.634404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.634380 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:02:02.639960 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:02.639935 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bef957b_baa4_4381_a556_cfa92982761f.slice/crio-a852b7e6731fe4617550fffc3b8d363d987f043be3bcd2ac1d4bebaaa5a9547f WatchSource:0}: Error finding container a852b7e6731fe4617550fffc3b8d363d987f043be3bcd2ac1d4bebaaa5a9547f: Status 404 returned error can't find the container with id a852b7e6731fe4617550fffc3b8d363d987f043be3bcd2ac1d4bebaaa5a9547f Nov 03 17:02:02.808515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.808485 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:02:02.808701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:02.808536 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:02:03.501843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.501802 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" event={"ID":"4bef957b-baa4-4381-a556-cfa92982761f","Type":"ContainerStarted","Data":"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d"} Nov 03 17:02:03.502047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.501863 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" event={"ID":"4bef957b-baa4-4381-a556-cfa92982761f","Type":"ContainerStarted","Data":"a852b7e6731fe4617550fffc3b8d363d987f043be3bcd2ac1d4bebaaa5a9547f"} Nov 03 17:02:03.502047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.501884 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:03.503509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.503479 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" event={"ID":"a38d94fd-440f-41d7-9a8b-1a3b42fc378a","Type":"ContainerStarted","Data":"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e"} Nov 03 17:02:03.503643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.503534 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:03.503643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.503555 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" event={"ID":"a38d94fd-440f-41d7-9a8b-1a3b42fc378a","Type":"ContainerStarted","Data":"bfec078242c73e38d5fdc77453234034a349547c2cd43528c6b5e17b21f5499e"} Nov 03 17:02:03.570871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.570799 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" podStartSLOduration=2.570780622 podStartE2EDuration="2.570780622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:03.56769721 +0000 UTC m=+2818.900910780" watchObservedRunningTime="2026-11-03 17:02:03.570780622 +0000 UTC m=+2818.903994193" Nov 03 17:02:03.611474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.611413 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" podStartSLOduration=2.611399138 podStartE2EDuration="2.611399138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:03.610317511 +0000 UTC m=+2818.943531080" watchObservedRunningTime="2026-11-03 17:02:03.611399138 +0000 UTC m=+2818.944612707" Nov 03 17:02:03.809237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.809143 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:02:03.809237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:03.809206 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:02:04.808213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:04.808182 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:02:04.808709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:04.808240 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:02:05.808821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:05.808785 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 17:02:05.809247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:05.808839 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 17:02:06.907292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:06.907254 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:02:06.908116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:06.908046 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:02:06.910644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:06.910612 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:02:10.566988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:10.566953 6001 scope.go:122] "RemoveContainer" containerID="5adc394badd20d85b23977de2e8b2593d98a72b74a254eaec251aceb9f6dd46e" Nov 03 17:02:14.514037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:14.514004 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:02:14.514439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:14.514081 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:02:21.012244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.012205 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:02:21.016619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.016593 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.035264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.035221 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:02:21.036199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.036167 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-fdbf0c\"" Nov 03 17:02:21.043344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.043315 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-fdbf0c-dockercfg-khvbw\"" Nov 03 17:02:21.064585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.064537 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:02:21.069398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.069372 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.088134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.088108 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:02:21.206465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.202570 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9frzb\" (UniqueName: \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.206465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.202812 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.206465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.202872 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blk2x\" (UniqueName: \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.206465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.202904 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.304158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.304053 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.304482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.304437 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blk2x\" (UniqueName: \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.304994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.304970 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.305334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.305313 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9frzb\" (UniqueName: \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.307033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.307010 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.308670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.308647 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.334856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.334823 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blk2x\" (UniqueName: \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\") pod \"payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.335262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.335237 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9frzb\" (UniqueName: \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\") pod \"payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.383543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.383512 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:21.632199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.632113 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:21.708979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.708554 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:02:21.977324 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:21.977294 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5684e128_aa30_430f_8604_8027740756cf.slice/crio-9829cedf4d9be05a9f6c2537bda24ed47283f6a550965942bf018099d8fbae69 WatchSource:0}: Error finding container 9829cedf4d9be05a9f6c2537bda24ed47283f6a550965942bf018099d8fbae69: Status 404 returned error can't find the container with id 9829cedf4d9be05a9f6c2537bda24ed47283f6a550965942bf018099d8fbae69 Nov 03 17:02:21.978365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:21.978336 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:02:22.253795 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:02:22.253706 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:02:22.701506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.701388 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" event={"ID":"5684e128-aa30-430f-8604-8027740756cf","Type":"ContainerStarted","Data":"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d"} Nov 03 17:02:22.701506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.701470 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:22.701506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.701492 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" event={"ID":"5684e128-aa30-430f-8604-8027740756cf","Type":"ContainerStarted","Data":"9829cedf4d9be05a9f6c2537bda24ed47283f6a550965942bf018099d8fbae69"} Nov 03 17:02:22.703187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.703156 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" event={"ID":"7996d93d-b563-40ff-baeb-0a6e3f8ba227","Type":"ContainerStarted","Data":"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4"} Nov 03 17:02:22.703333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.703204 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" event={"ID":"7996d93d-b563-40ff-baeb-0a6e3f8ba227","Type":"ContainerStarted","Data":"ae925cf709942f0b171aec964d84b542dacd0f3d0faaa77e4acafca22806725c"} Nov 03 17:02:22.703333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.703223 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:22.747609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.747547 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" podStartSLOduration=2.747533933 podStartE2EDuration="2.747533933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:22.744802552 +0000 UTC m=+2838.078016114" watchObservedRunningTime="2026-11-03 17:02:22.747533933 +0000 UTC m=+2838.080747501" Nov 03 17:02:22.788844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:22.788776 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" podStartSLOduration=1.788759125 podStartE2EDuration="1.788759125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:22.787349747 +0000 UTC m=+2838.120563319" watchObservedRunningTime="2026-11-03 17:02:22.788759125 +0000 UTC m=+2838.121972693" Nov 03 17:02:23.437818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.437777 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:02:23.442196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.442168 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.457630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.457601 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:02:23.461503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.461467 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.464851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.464829 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1f83c0-openshift-default-dockercfg-bw8dj\"" Nov 03 17:02:23.470694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.470655 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:02:23.500802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.500764 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:02:23.556029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.555987 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556123 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.556226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556174 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556268 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.556351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556327 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgcmd\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556354 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556380 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556533 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556615 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.556675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556663 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.556735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556683 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmt7\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.556859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556769 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556823 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556876 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.556952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556934 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.557019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556969 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.557019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.556985 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.557019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.557004 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.658842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.658793 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.658886 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.658952 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.658998 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659035 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659067 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659131 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659197 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659233 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659277 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659303 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgcmd\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659334 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.659376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659360 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659399 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659418 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659591 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659656 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659674 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmt7\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.660047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.659710 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.660288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.660245 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.660288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.660254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.660574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.660543 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.661134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.661106 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.661395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.661338 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.661395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.661377 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.661635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.661619 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.661790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.661774 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.662062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.662044 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.663267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.663244 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.663379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.663357 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.663970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.663921 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.664382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.664361 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.683883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.683854 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.684421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.684397 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmt7\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:23.684662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.684643 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.685313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.685291 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgcmd\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.759757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.759722 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:23.784452 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:23.784405 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:24.088837 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:24.088807 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158091eb_b54f_4d18_8388_1904b664d3c1.slice/crio-09f44e351de2262f5310d776e98d39c19d1ebb46a882a5170a0253d39c21fb63 WatchSource:0}: Error finding container 09f44e351de2262f5310d776e98d39c19d1ebb46a882a5170a0253d39c21fb63: Status 404 returned error can't find the container with id 09f44e351de2262f5310d776e98d39c19d1ebb46a882a5170a0253d39c21fb63 Nov 03 17:02:24.090494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.090431 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:02:24.091799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.091769 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.091889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.091830 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.091889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.091858 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.152225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.152198 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:02:24.153887 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:24.153860 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3756ab63_6407_41c4_8b8b_8596cb24e43b.slice/crio-0f70b4e13491821475b0c63a4134856491029288a369ed0bba4e499680b0745c WatchSource:0}: Error finding container 0f70b4e13491821475b0c63a4134856491029288a369ed0bba4e499680b0745c: Status 404 returned error can't find the container with id 0f70b4e13491821475b0c63a4134856491029288a369ed0bba4e499680b0745c Nov 03 17:02:24.157331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.157296 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.157403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.157379 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.157459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.157420 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:24.731487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.730392 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" event={"ID":"3756ab63-6407-41c4-8b8b-8596cb24e43b","Type":"ContainerStarted","Data":"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d"} Nov 03 17:02:24.731487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.730477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" event={"ID":"3756ab63-6407-41c4-8b8b-8596cb24e43b","Type":"ContainerStarted","Data":"0f70b4e13491821475b0c63a4134856491029288a369ed0bba4e499680b0745c"} Nov 03 17:02:24.737433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.736524 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" event={"ID":"158091eb-b54f-4d18-8388-1904b664d3c1","Type":"ContainerStarted","Data":"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791"} Nov 03 17:02:24.737433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.736606 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" event={"ID":"158091eb-b54f-4d18-8388-1904b664d3c1","Type":"ContainerStarted","Data":"09f44e351de2262f5310d776e98d39c19d1ebb46a882a5170a0253d39c21fb63"} Nov 03 17:02:24.761616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.761531 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:24.823606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.823559 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:24.914469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.914042 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podStartSLOduration=1.91402622 podStartE2EDuration="1.91402622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:24.802828022 +0000 UTC m=+2840.136041593" watchObservedRunningTime="2026-11-03 17:02:24.91402622 +0000 UTC m=+2840.247239791" Nov 03 17:02:24.916153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:24.916017 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podStartSLOduration=1.9160054290000001 podStartE2EDuration="1.916005429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:24.912674847 +0000 UTC m=+2840.245888442" watchObservedRunningTime="2026-11-03 17:02:24.916005429 +0000 UTC m=+2840.249219000" Nov 03 17:02:25.794471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:25.793580 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:02:25.794471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:25.793648 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Nov 03 17:02:25.795117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:25.794914 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:02:25.795117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:25.794957 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 03 17:02:26.773470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:26.769895 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:02:26.773470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:26.769969 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Nov 03 17:02:26.801516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:26.799674 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:02:26.801516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:26.799764 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 03 17:02:27.782087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:27.782048 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:02:27.782472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:27.782415 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Nov 03 17:02:27.786842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:27.786813 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:02:27.787030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:27.787003 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 03 17:02:28.771470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:28.770982 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:02:28.771470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:28.771049 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Nov 03 17:02:28.799496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:28.792326 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:02:28.799496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:28.792390 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 03 17:02:29.769469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:29.768971 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:02:29.769469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:29.769061 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Nov 03 17:02:29.789470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:29.785849 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:02:29.789470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:29.785914 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.14:15021/healthz/ready\": dial tcp 10.134.1.14:15021: connect: connection refused" Nov 03 17:02:30.765643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.765612 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:30.766101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.766081 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:30.766837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.766819 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:02:30.791604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.791571 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:30.791997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.791852 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:30.792654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:30.792634 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:02:33.715508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:33.715477 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:02:33.715995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:33.715556 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:02:51.095407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.095367 6001 generic.go:356] "Generic (PLEG): container finished" podID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerID="3a24a22b4f97b88583ad6a11c3d4606d9d43d342609e3c73ac2e9ba019d68fe0" exitCode=0 Nov 03 17:02:51.095911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.095478 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerDied","Data":"3a24a22b4f97b88583ad6a11c3d4606d9d43d342609e3c73ac2e9ba019d68fe0"} Nov 03 17:02:51.096008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.095993 6001 scope.go:122] "RemoveContainer" containerID="3a24a22b4f97b88583ad6a11c3d4606d9d43d342609e3c73ac2e9ba019d68fe0" Nov 03 17:02:51.098086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.098054 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:51.098169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.098131 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:51.098169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:51.098165 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:52.107296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:52.107253 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerStarted","Data":"d837c3bce145c6d7556f06d15bc8449db9a9599a343fec16a6d52c65bccba89b"} Nov 03 17:02:52.255248 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:02:52.255204 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:02:52.687651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:52.687610 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:02:52.689407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:52.689374 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:52.689632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:52.689428 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:53.102664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.102611 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:02:53.579023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.578982 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:02:53.583916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.583891 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.628580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.626846 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:02:53.682610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682557 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.682841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682633 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.682841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682761 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.682985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682857 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.682985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682914 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.682985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.682944 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.683164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.683066 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jhkb\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.683164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.683123 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.683279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.683173 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.688263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.688226 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:53.688438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.688290 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:53.784626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784578 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jhkb\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.784845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784642 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.784845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784679 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.784845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784783 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.784845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784834 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784865 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.784959 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785009 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785046 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785200 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785384 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785519 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785565 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.785648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.785634 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.787757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.787732 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.787938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.787920 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.809105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.809066 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.809258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.809213 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jhkb\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:53.903073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:53.902971 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:54.273626 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:02:54.273576 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05c6d79_8277_491b_a483_d5521eabbe31.slice/crio-9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b WatchSource:0}: Error finding container 9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b: Status 404 returned error can't find the container with id 9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b Nov 03 17:02:54.275275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.275243 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:02:54.277220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.277173 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:54.277331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.277253 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:54.277331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.277283 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:02:54.687719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.687626 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:54.687719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:54.687698 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:55.157033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.156984 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" event={"ID":"b05c6d79-8277-491b-a483-d5521eabbe31","Type":"ContainerStarted","Data":"e13b056792daba1e57c0a1e63157810dadd9c6fed03797b1e8b2a9820962e2dd"} Nov 03 17:02:55.170183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.170131 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" event={"ID":"b05c6d79-8277-491b-a483-d5521eabbe31","Type":"ContainerStarted","Data":"9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b"} Nov 03 17:02:55.221411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.221342 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podStartSLOduration=2.221323977 podStartE2EDuration="2.221323977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:55.218591073 +0000 UTC m=+2870.551804657" watchObservedRunningTime="2026-11-03 17:02:55.221323977 +0000 UTC m=+2870.554537546" Nov 03 17:02:55.688435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.688393 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:55.688870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.688478 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:55.903371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.903329 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:02:55.906483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.906430 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:02:55.906683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:55.906537 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:02:56.691473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:56.689384 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:56.691473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:56.689552 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:56.903802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:56.903763 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:02:56.904103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:56.904074 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:02:57.689269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:57.688505 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" start-of-body= Nov 03 17:02:57.689269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:57.688580 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.6:15021/healthz/ready\": dial tcp 10.134.1.6:15021: connect: connection refused" Nov 03 17:02:57.903347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:57.903303 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:02:57.903882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:57.903371 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:02:58.693733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:58.693700 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:02:58.694793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:58.694769 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:02:58.903912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:58.903860 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:02:58.904311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:58.903943 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:02:59.903980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:59.903944 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:02:59.904416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:02:59.904000 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:03:00.904291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:00.904251 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 17:03:00.904772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:00.904328 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 17:03:01.908915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:01.908883 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:03:01.909437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:01.909418 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:03:01.910464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:01.910428 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:03:03.344157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.344111 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:03:03.349133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.349099 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.372516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.372481 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-790111-openshift-default-dockercfg-kp66f\"" Nov 03 17:03:03.377978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.377941 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:03:03.378486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378422 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378499 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378561 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdtjb\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378670 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378785 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378861 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.378942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378935 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.379065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.378987 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.379065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.379047 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481371 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481437 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481512 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481597 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481692 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdtjb\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481806 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481835 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.481953 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.482176 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.482735 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.482891 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.484467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.483165 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.497537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.489930 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.551500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.551468 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdtjb\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.564490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.560093 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.582479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.577595 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\") pod \"e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:03.680049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:03.677132 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:04.276169 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:03:04.276137 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7383014e_2260_45d8_ae51_cfc249d6a476.slice/crio-5fe49523634c20630379fe95540814180ad6980786b5345b00964165f8a912ca WatchSource:0}: Error finding container 5fe49523634c20630379fe95540814180ad6980786b5345b00964165f8a912ca: Status 404 returned error can't find the container with id 5fe49523634c20630379fe95540814180ad6980786b5345b00964165f8a912ca Nov 03 17:03:04.279343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.279307 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:04.279419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.279388 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:04.279508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.279421 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:04.288325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.288292 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:03:04.399147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.399102 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerStarted","Data":"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b"} Nov 03 17:03:04.399637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.399169 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerStarted","Data":"5fe49523634c20630379fe95540814180ad6980786b5345b00964165f8a912ca"} Nov 03 17:03:04.465505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.460545 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podStartSLOduration=1.460519943 podStartE2EDuration="1.460519943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:04.456856193 +0000 UTC m=+2879.790069765" watchObservedRunningTime="2026-11-03 17:03:04.460519943 +0000 UTC m=+2879.793733515" Nov 03 17:03:04.678243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:04.678144 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:05.684438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:05.684407 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:06.414459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:06.414424 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:06.415789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:06.415760 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:03:09.107468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.103997 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:03:09.132469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.130867 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.165468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.165172 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-790111\"" Nov 03 17:03:09.225624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.224664 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmj9f\" (UniqueName: \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.225624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.224879 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.231670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.226363 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-790111-dockercfg-57rvb\"" Nov 03 17:03:09.324369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.324339 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:03:09.326736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.326708 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmj9f\" (UniqueName: \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.327118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.327094 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.328313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.328070 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.360536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.360432 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmj9f\" (UniqueName: \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\") pod \"payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:09.646608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:09.640923 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:10.142621 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:03:10.142583 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5567754_676a_4298_9dd7_50ccfab12a68.slice/crio-eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981 WatchSource:0}: Error finding container eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981: Status 404 returned error can't find the container with id eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981 Nov 03 17:03:10.143842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:10.143810 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:03:10.450044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:10.449947 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" event={"ID":"d5567754-676a-4298-9dd7-50ccfab12a68","Type":"ContainerStarted","Data":"0a7ea6f8f3a450936cf47270434599148c19b614464d2ab9e3ee923158bb8239"} Nov 03 17:03:10.450044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:10.450012 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" event={"ID":"d5567754-676a-4298-9dd7-50ccfab12a68","Type":"ContainerStarted","Data":"eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981"} Nov 03 17:03:10.450044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:10.450044 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:10.497131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:10.497063 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" podStartSLOduration=2.4970459959999998 podStartE2EDuration="2.497045996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:10.49274774 +0000 UTC m=+2885.825961310" watchObservedRunningTime="2026-11-03 17:03:10.497045996 +0000 UTC m=+2885.830259609" Nov 03 17:03:11.161859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.161814 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:03:11.166134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.166104 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.185505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.185468 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:03:11.196500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.194730 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.211562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.211526 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:03:11.211754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.211605 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e0e63d\"" Nov 03 17:03:11.211754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.211688 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"openshift-service-ca.crt\"" Nov 03 17:03:11.211754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.211697 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"kube-root-ca.crt\"" Nov 03 17:03:11.215877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.213681 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"routing-overlay\"" Nov 03 17:03:11.218260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.218212 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:03:11.230646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.230609 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ca9d7c-openshift-default-dockercfg-fgrm2\"" Nov 03 17:03:11.230855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.230652 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-0db41f4e4f64\"" Nov 03 17:03:11.230855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.230694 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-e2e--2172597b-dockercfg-zcksp\"" Nov 03 17:03:11.289556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289518 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.289744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289585 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.289744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289687 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.289744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289718 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.289865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289837 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.289903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289880 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.290005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.289989 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.290098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290085 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.290172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290160 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.290209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290182 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjbx\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.290245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290224 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.290290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290273 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.290328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.290304 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.396797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.396741 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.397971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.397943 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.398235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.398213 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.398512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.398490 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.398739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.398718 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.398884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.398867 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjbx\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.410789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.410758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.411017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.410997 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.411141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.411126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.411302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.411283 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.411497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.411479 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.411660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.411646 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.411775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.411762 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.412252 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.397863 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.401875 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.415787 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.402105 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.416422 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.427512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.416676 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.431838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.431799 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.433093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.433066 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.438828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.438800 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjbx\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.444581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.444504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.460764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.452088 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.465464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.461393 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.487471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.483028 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:11.542046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.528542 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:11.901792 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:03:11.901756 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode423f74e_8f1c_41aa_982b_3f58a244d352.slice/crio-d9cae091ccccc986054dbb79f00812187d6d006cdf0cee491ceec24346be15b0 WatchSource:0}: Error finding container d9cae091ccccc986054dbb79f00812187d6d006cdf0cee491ceec24346be15b0: Status 404 returned error can't find the container with id d9cae091ccccc986054dbb79f00812187d6d006cdf0cee491ceec24346be15b0 Nov 03 17:03:11.902534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.902506 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:03:11.983513 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:03:11.983434 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64 WatchSource:0}: Error finding container bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64: Status 404 returned error can't find the container with id bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64 Nov 03 17:03:11.984324 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.984293 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:03:11.986749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.986716 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:11.986832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.986786 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:11.986832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:11.986815 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:03:12.478931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.478893 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" event={"ID":"8678db30-57d9-402a-9b9e-f05362e0ef2a","Type":"ContainerStarted","Data":"a51c1b2c6356c21ca77a573fab6f3abe7a19a5c78a217cd95f71a23c315e5356"} Nov 03 17:03:12.479497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.478963 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" event={"ID":"8678db30-57d9-402a-9b9e-f05362e0ef2a","Type":"ContainerStarted","Data":"bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64"} Nov 03 17:03:12.480978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.480948 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" event={"ID":"e423f74e-8f1c-41aa-982b-3f58a244d352","Type":"ContainerStarted","Data":"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601"} Nov 03 17:03:12.481122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.480998 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" event={"ID":"e423f74e-8f1c-41aa-982b-3f58a244d352","Type":"ContainerStarted","Data":"d9cae091ccccc986054dbb79f00812187d6d006cdf0cee491ceec24346be15b0"} Nov 03 17:03:12.481122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.481041 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:12.484220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.484202 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:12.532689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.532622 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podStartSLOduration=1.532601379 podStartE2EDuration="1.532601379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:12.529617698 +0000 UTC m=+2887.862831270" watchObservedRunningTime="2026-11-03 17:03:12.532601379 +0000 UTC m=+2887.865814949" Nov 03 17:03:12.576606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:12.576527 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" podStartSLOduration=1.576506417 podStartE2EDuration="1.576506417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:12.573742094 +0000 UTC m=+2887.906955663" watchObservedRunningTime="2026-11-03 17:03:12.576506417 +0000 UTC m=+2887.909719988" Nov 03 17:03:13.485055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:13.485022 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:13.485601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:13.485079 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:14.484046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:14.484008 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:14.484233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:14.484064 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:15.483953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:15.483909 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:15.484493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:15.483980 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:16.484717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:16.484629 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:16.484717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:16.484687 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:17.484062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:17.484023 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:17.484254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:17.484087 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:18.484269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:18.484233 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Nov 03 17:03:18.484719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:18.484290 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Nov 03 17:03:19.490002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:19.489965 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:19.490562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:19.490461 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:19.491207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:19.491184 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:03:19.739797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:19.739759 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 17:03:19.740255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:19.740154 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" containerID="cri-o://a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418" gracePeriod=30 Nov 03 17:03:21.470820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:21.470787 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:03:22.252515 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:03:22.252473 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:03:23.493775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:23.493746 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:03:24.054805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:24.054768 6001 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:03:24.054997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:24.054848 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:03:25.072416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.072380 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 17:03:25.217028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.216981 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217100 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217196 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217305 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217315 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:25.217542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217400 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217541 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217598 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217720 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217774 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\") pod \"4b6d9e26-425a-4478-9b90-6e5a9098626c\" (UID: \"4b6d9e26-425a-4478-9b90-6e5a9098626c\") " Nov 03 17:03:25.217829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.217788 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:25.218180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218048 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218275 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218821 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218848 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218864 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218878 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.219019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.218871 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:03:25.224005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.220855 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:25.224005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.221330 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "kube-api-access-zr2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:25.224005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.221813 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:25.224005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.221899 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo" pod "4b6d9e26-425a-4478-9b90-6e5a9098626c" (UID: "4b6d9e26-425a-4478-9b90-6e5a9098626c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:03:25.321033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.320924 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b6d9e26-425a-4478-9b90-6e5a9098626c-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.321033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.320962 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2km\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-kube-api-access-zr2km\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.321033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.320975 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.321033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.320986 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.321033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.320997 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b6d9e26-425a-4478-9b90-6e5a9098626c-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:03:25.607920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.607805 6001 generic.go:356] "Generic (PLEG): container finished" podID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerID="a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418" exitCode=0 Nov 03 17:03:25.608105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.607993 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" event={"ID":"4b6d9e26-425a-4478-9b90-6e5a9098626c","Type":"ContainerDied","Data":"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418"} Nov 03 17:03:25.608105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.608087 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" event={"ID":"4b6d9e26-425a-4478-9b90-6e5a9098626c","Type":"ContainerDied","Data":"2e7975464a1c0961684577c4f91df59c0565a95d525e8b1cecb03fcfbc44355e"} Nov 03 17:03:25.608224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.608126 6001 scope.go:122] "RemoveContainer" containerID="a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418" Nov 03 17:03:25.608624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.608599 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6" Nov 03 17:03:25.626120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.626092 6001 scope.go:122] "RemoveContainer" containerID="a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418" Nov 03 17:03:25.626716 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:03:25.626679 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418\": container with ID starting with a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418 not found: ID does not exist" containerID="a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418" Nov 03 17:03:25.626935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.626896 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418"} err="failed to get container status \"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418\": rpc error: code = NotFound desc = could not find container \"a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418\": container with ID starting with a1b2574b7a6a5ce2f1c50c6678f70e3ef24f276a18412c971f8e65558cb1b418 not found: ID does not exist" Nov 03 17:03:25.700192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.700150 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 17:03:25.705279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:25.705234 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-lbdq6"] Nov 03 17:03:26.793042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:26.793003 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" path="/var/lib/kubelet/pods/4b6d9e26-425a-4478-9b90-6e5a9098626c/volumes" Nov 03 17:03:36.032631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.032592 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:03:36.033691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.033667 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" Nov 03 17:03:36.033691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.033691 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" Nov 03 17:03:36.033983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.033966 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b6d9e26-425a-4478-9b90-6e5a9098626c" containerName="istio-proxy" Nov 03 17:03:36.037203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.037181 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.058358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.058320 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:03:36.063706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.063671 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.063880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.063819 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.064569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.063946 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.064569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.063985 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.066405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.066361 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"kube-root-ca.crt\"" Nov 03 17:03:36.066578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.066494 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-4ce9c9bbf166\"" Nov 03 17:03:36.066578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.066546 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-e2e--d0a79461-dockercfg-c9d8c\"" Nov 03 17:03:36.066890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.066869 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"routing-overlay\"" Nov 03 17:03:36.066994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.066901 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"openshift-service-ca.crt\"" Nov 03 17:03:36.067120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.067083 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ca9d7c\"" Nov 03 17:03:36.165083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.165041 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.165269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.165135 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.165269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.165201 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.165269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.165223 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.165995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.165969 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.166078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.166036 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.167805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.167781 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.201467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.191375 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.355529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.352745 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:36.754090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.754059 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:03:36.755375 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:03:36.755342 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3514af24_b455_43b0_8840_9ab947e57ec1.slice/crio-882905f9810bc93e799e6d31eef7bb6d2bcdd23aefefd2dfd35d4a21c3e09e36 WatchSource:0}: Error finding container 882905f9810bc93e799e6d31eef7bb6d2bcdd23aefefd2dfd35d4a21c3e09e36: Status 404 returned error can't find the container with id 882905f9810bc93e799e6d31eef7bb6d2bcdd23aefefd2dfd35d4a21c3e09e36 Nov 03 17:03:36.798474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:36.798423 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" event={"ID":"3514af24-b455-43b0-8840-9ab947e57ec1","Type":"ContainerStarted","Data":"882905f9810bc93e799e6d31eef7bb6d2bcdd23aefefd2dfd35d4a21c3e09e36"} Nov 03 17:03:37.800652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:37.800611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" event={"ID":"3514af24-b455-43b0-8840-9ab947e57ec1","Type":"ContainerStarted","Data":"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed"} Nov 03 17:03:37.801168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:37.800705 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:37.876465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:37.871730 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" podStartSLOduration=1.871707896 podStartE2EDuration="1.871707896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:37.855908723 +0000 UTC m=+2913.189122319" watchObservedRunningTime="2026-11-03 17:03:37.871707896 +0000 UTC m=+2913.204921467" Nov 03 17:03:48.818080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:03:48.818043 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:03:52.296960 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:03:52.296919 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:04:14.053303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:14.053262 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58"] Nov 03 17:04:14.065916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:14.065877 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38680f90-lmz58"] Nov 03 17:04:14.829517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:14.828538 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a548ba-0060-4691-82b0-b9fe4e050182" path="/var/lib/kubelet/pods/f6a548ba-0060-4691-82b0-b9fe4e050182/volumes" Nov 03 17:04:22.276408 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:04:22.276368 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:04:24.034314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:24.034276 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x"] Nov 03 17:04:24.049436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:24.049394 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b024d73-lfd6x"] Nov 03 17:04:24.795150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:24.795118 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5a3d56-d0c0-4199-b5c7-44439436b279" path="/var/lib/kubelet/pods/1e5a3d56-d0c0-4199-b5c7-44439436b279/volumes" Nov 03 17:04:26.363583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.363546 6001 generic.go:356] "Generic (PLEG): container finished" podID="7383014e-2260-45d8-ae51-cfc249d6a476" containerID="1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b" exitCode=0 Nov 03 17:04:26.364119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.363620 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerDied","Data":"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b"} Nov 03 17:04:26.364119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.364099 6001 scope.go:122] "RemoveContainer" containerID="1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b" Nov 03 17:04:26.366050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.366023 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:04:26.366135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.366092 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:04:26.366180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:26.366144 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:04:27.374696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:27.374655 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerStarted","Data":"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c"} Nov 03 17:04:27.680522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:27.677917 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:04:27.680522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:27.678048 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:27.680522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:27.678105 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:28.677911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:28.677864 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:28.678431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:28.677941 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:29.677989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:29.677952 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:29.678514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:29.678011 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:30.678059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:30.678017 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:30.678580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:30.678082 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:31.678179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:31.678128 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:31.678752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:31.678205 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:32.677941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:32.677901 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:04:32.678174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:32.677964 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 17:04:33.709898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:33.709858 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:04:33.712323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:33.712297 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:04:33.715512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:33.715490 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:04:41.983439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.983396 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:04:41.987407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.987385 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:41.999456 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.999415 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:41.999638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.999615 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd9c6\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:41.999721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.999709 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:41.999874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:41.999860 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.009229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.009190 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:04:42.015653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.015610 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-1faf98cb4c54\"" Nov 03 17:04:42.101402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.101358 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.101614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.101591 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.101753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.101734 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.101924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.101904 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd9c6\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.102215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.102162 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.102469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.102432 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.104338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.104308 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.126120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.126093 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd9c6\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.301992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.301902 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:42.611353 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:04:42.611319 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb462b489_4dd9_4c29_9ba9_cc60c24c1d19.slice/crio-554a165d39477855b8e53bde37279108d29f019f0850cf5560dec9d11cb7987e WatchSource:0}: Error finding container 554a165d39477855b8e53bde37279108d29f019f0850cf5560dec9d11cb7987e: Status 404 returned error can't find the container with id 554a165d39477855b8e53bde37279108d29f019f0850cf5560dec9d11cb7987e Nov 03 17:04:42.612378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:42.612351 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:04:43.533285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:43.533242 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" event={"ID":"b462b489-4dd9-4c29-9ba9-cc60c24c1d19","Type":"ContainerStarted","Data":"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0"} Nov 03 17:04:43.533717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:43.533306 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" event={"ID":"b462b489-4dd9-4c29-9ba9-cc60c24c1d19","Type":"ContainerStarted","Data":"554a165d39477855b8e53bde37279108d29f019f0850cf5560dec9d11cb7987e"} Nov 03 17:04:43.533717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:43.533336 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:43.581101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:43.581036 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" podStartSLOduration=2.5810226419999998 podStartE2EDuration="2.581022642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:04:43.576814444 +0000 UTC m=+2978.910028013" watchObservedRunningTime="2026-11-03 17:04:43.581022642 +0000 UTC m=+2978.914236205" Nov 03 17:04:52.255217 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:04:52.255137 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:04:53.928265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:53.928230 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:04:54.545397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:54.545365 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:04:54.634882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:54.634848 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:04:58.929563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:58.929515 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" podUID="3514af24-b455-43b0-8840-9ab947e57ec1" containerName="payload-processing" containerID="cri-o://3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed" gracePeriod=25 Nov 03 17:04:59.245273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.245242 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:04:59.298165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.298131 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\") pod \"3514af24-b455-43b0-8840-9ab947e57ec1\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " Nov 03 17:04:59.298347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.298227 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\") pod \"3514af24-b455-43b0-8840-9ab947e57ec1\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " Nov 03 17:04:59.298347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.298288 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\") pod \"3514af24-b455-43b0-8840-9ab947e57ec1\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " Nov 03 17:04:59.298494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.298462 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\") pod \"3514af24-b455-43b0-8840-9ab947e57ec1\" (UID: \"3514af24-b455-43b0-8840-9ab947e57ec1\") " Nov 03 17:04:59.298724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.298689 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume" pod "3514af24-b455-43b0-8840-9ab947e57ec1" (UID: "3514af24-b455-43b0-8840-9ab947e57ec1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:04:59.299193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.299164 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume" pod "3514af24-b455-43b0-8840-9ab947e57ec1" (UID: "3514af24-b455-43b0-8840-9ab947e57ec1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:04:59.299393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.299375 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:04:59.305048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.305010 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s" pod "3514af24-b455-43b0-8840-9ab947e57ec1" (UID: "3514af24-b455-43b0-8840-9ab947e57ec1"). InnerVolumeSpecName "kube-api-access-fqj9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:04:59.305192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.305113 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials" pod "3514af24-b455-43b0-8840-9ab947e57ec1" (UID: "3514af24-b455-43b0-8840-9ab947e57ec1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:04:59.439665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.439628 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3514af24-b455-43b0-8840-9ab947e57ec1-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:04:59.440353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.440331 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:04:59.440546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.440530 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqj9s\" (UniqueName: \"kubernetes.io/projected/3514af24-b455-43b0-8840-9ab947e57ec1-kube-api-access-fqj9s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:04:59.635761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.635690 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" podUID="e423f74e-8f1c-41aa-982b-3f58a244d352" containerName="payload-processing" containerID="cri-o://8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601" gracePeriod=25 Nov 03 17:04:59.682908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.682874 6001 generic.go:356] "Generic (PLEG): container finished" podID="3514af24-b455-43b0-8840-9ab947e57ec1" containerID="3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed" exitCode=0 Nov 03 17:04:59.683093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.682928 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" Nov 03 17:04:59.683093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.682959 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" event={"ID":"3514af24-b455-43b0-8840-9ab947e57ec1","Type":"ContainerDied","Data":"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed"} Nov 03 17:04:59.683093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.683019 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x" event={"ID":"3514af24-b455-43b0-8840-9ab947e57ec1","Type":"ContainerDied","Data":"882905f9810bc93e799e6d31eef7bb6d2bcdd23aefefd2dfd35d4a21c3e09e36"} Nov 03 17:04:59.683093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.683049 6001 scope.go:122] "RemoveContainer" containerID="3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed" Nov 03 17:04:59.754439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.754337 6001 scope.go:122] "RemoveContainer" containerID="3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed" Nov 03 17:04:59.755135 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:04:59.755046 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed\": container with ID starting with 3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed not found: ID does not exist" containerID="3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed" Nov 03 17:04:59.755135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.755088 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed"} err="failed to get container status \"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed\": rpc error: code = NotFound desc = could not find container \"3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed\": container with ID starting with 3563d01cc6bea17aeb589ffc781015c02e33a1da8c243263eaa525a19a74c0ed not found: ID does not exist" Nov 03 17:04:59.759632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.759605 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:04:59.772171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.772134 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8cwmb4x"] Nov 03 17:04:59.901494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:04:59.901467 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:05:00.051782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.051745 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\") pod \"e423f74e-8f1c-41aa-982b-3f58a244d352\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " Nov 03 17:05:00.052298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.051855 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\") pod \"e423f74e-8f1c-41aa-982b-3f58a244d352\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " Nov 03 17:05:00.052298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.051958 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\") pod \"e423f74e-8f1c-41aa-982b-3f58a244d352\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " Nov 03 17:05:00.052298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.052059 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\") pod \"e423f74e-8f1c-41aa-982b-3f58a244d352\" (UID: \"e423f74e-8f1c-41aa-982b-3f58a244d352\") " Nov 03 17:05:00.052298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.052191 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume" pod "e423f74e-8f1c-41aa-982b-3f58a244d352" (UID: "e423f74e-8f1c-41aa-982b-3f58a244d352"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:00.052608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.052519 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume" pod "e423f74e-8f1c-41aa-982b-3f58a244d352" (UID: "e423f74e-8f1c-41aa-982b-3f58a244d352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:00.052976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.052956 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:00.053037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.052982 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e423f74e-8f1c-41aa-982b-3f58a244d352-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:00.054501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.054472 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials" pod "e423f74e-8f1c-41aa-982b-3f58a244d352" (UID: "e423f74e-8f1c-41aa-982b-3f58a244d352"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:00.054637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.054521 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7" pod "e423f74e-8f1c-41aa-982b-3f58a244d352" (UID: "e423f74e-8f1c-41aa-982b-3f58a244d352"). InnerVolumeSpecName "kube-api-access-z9jf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:00.155489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.155362 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jf7\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-kube-api-access-z9jf7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:00.155489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.155395 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e423f74e-8f1c-41aa-982b-3f58a244d352-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:00.695018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.694976 6001 generic.go:356] "Generic (PLEG): container finished" podID="e423f74e-8f1c-41aa-982b-3f58a244d352" containerID="8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601" exitCode=0 Nov 03 17:05:00.695227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.695027 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" Nov 03 17:05:00.695227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.695055 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" event={"ID":"e423f74e-8f1c-41aa-982b-3f58a244d352","Type":"ContainerDied","Data":"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601"} Nov 03 17:05:00.695227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.695118 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx" event={"ID":"e423f74e-8f1c-41aa-982b-3f58a244d352","Type":"ContainerDied","Data":"d9cae091ccccc986054dbb79f00812187d6d006cdf0cee491ceec24346be15b0"} Nov 03 17:05:00.695227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.695141 6001 scope.go:122] "RemoveContainer" containerID="8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601" Nov 03 17:05:00.707516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.707492 6001 scope.go:122] "RemoveContainer" containerID="8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601" Nov 03 17:05:00.707860 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:00.707836 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601\": container with ID starting with 8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601 not found: ID does not exist" containerID="8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601" Nov 03 17:05:00.707970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.707870 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601"} err="failed to get container status \"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601\": rpc error: code = NotFound desc = could not find container \"8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601\": container with ID starting with 8becd618d38b06162580b0a463e47939bebfd7274e64bd41ab4e5985554bc601 not found: ID does not exist" Nov 03 17:05:00.752064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.752029 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:05:00.764510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.764467 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4zd7tx"] Nov 03 17:05:00.791752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.791717 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3514af24-b455-43b0-8840-9ab947e57ec1" path="/var/lib/kubelet/pods/3514af24-b455-43b0-8840-9ab947e57ec1/volumes" Nov 03 17:05:00.792325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:00.792308 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e423f74e-8f1c-41aa-982b-3f58a244d352" path="/var/lib/kubelet/pods/e423f74e-8f1c-41aa-982b-3f58a244d352/volumes" Nov 03 17:05:01.546211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.546162 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:05:01.547292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547265 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e423f74e-8f1c-41aa-982b-3f58a244d352" containerName="payload-processing" Nov 03 17:05:01.547292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547293 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e423f74e-8f1c-41aa-982b-3f58a244d352" containerName="payload-processing" Nov 03 17:05:01.547490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547330 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3514af24-b455-43b0-8840-9ab947e57ec1" containerName="payload-processing" Nov 03 17:05:01.547490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547339 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3514af24-b455-43b0-8840-9ab947e57ec1" containerName="payload-processing" Nov 03 17:05:01.547606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547581 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3514af24-b455-43b0-8840-9ab947e57ec1" containerName="payload-processing" Nov 03 17:05:01.547657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.547620 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e423f74e-8f1c-41aa-982b-3f58a244d352" containerName="payload-processing" Nov 03 17:05:01.551785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.551762 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.571343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.571311 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-6046c2-openshift-default-dockercfg-47pst\"" Nov 03 17:05:01.607507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.595365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:05:01.703487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.701716 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.718820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.718690 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.719100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.718903 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.719226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.719203 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.719306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.719279 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.719368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.719307 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24k8w\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.731753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.731718 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.732024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.732002 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.732184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.732171 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835081 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24k8w\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835294 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835339 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835421 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835475 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835568 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835640 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835643 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835714 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835810 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.835954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.835911 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.836083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.836039 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.836760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.836731 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.837916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.837891 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.838172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.838153 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.859191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.859157 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.859295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.859278 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24k8w\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\") pod \"e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:01.866225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:01.866203 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:02.185897 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:02.185867 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17eaa40_3d12_4f36_bdee_e4c44f657e11.slice/crio-1dc8f136bcafba2086d504dd85c933d51479a8e00a9a449b74966820092c1c11 WatchSource:0}: Error finding container 1dc8f136bcafba2086d504dd85c933d51479a8e00a9a449b74966820092c1c11: Status 404 returned error can't find the container with id 1dc8f136bcafba2086d504dd85c933d51479a8e00a9a449b74966820092c1c11 Nov 03 17:05:02.187601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.187497 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:05:02.188891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.188857 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:02.188971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.188931 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:02.188971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.188959 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:02.743791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.743756 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" event={"ID":"b17eaa40-3d12-4f36-bdee-e4c44f657e11","Type":"ContainerStarted","Data":"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384"} Nov 03 17:05:02.744238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.743816 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" event={"ID":"b17eaa40-3d12-4f36-bdee-e4c44f657e11","Type":"ContainerStarted","Data":"1dc8f136bcafba2086d504dd85c933d51479a8e00a9a449b74966820092c1c11"} Nov 03 17:05:02.794111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.794062 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podStartSLOduration=1.7940483870000001 podStartE2EDuration="1.794048387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:02.79349083 +0000 UTC m=+2998.126704395" watchObservedRunningTime="2026-11-03 17:05:02.794048387 +0000 UTC m=+2998.127261955" Nov 03 17:05:02.866821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:02.866785 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:03.870707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:03.870655 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 03 17:05:03.871252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:03.870726 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 03 17:05:04.869853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:04.869814 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 03 17:05:04.870077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:04.869882 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 03 17:05:05.372435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:05.372406 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:05:05.376517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:05.376491 6001 kubelet.go:1689] "Image garbage collection succeeded" Nov 03 17:05:05.489126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:05.489097 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:05:05.868878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:05.868838 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 03 17:05:05.869134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:05.868903 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 03 17:05:06.868982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:06.868936 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 03 17:05:06.869555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:06.869008 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 03 17:05:07.868920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:07.868876 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" start-of-body= Nov 03 17:05:07.869127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:07.868941 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.22:15021/healthz/ready\": dial tcp 10.134.1.22:15021: connect: connection refused" Nov 03 17:05:09.903469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:09.902710 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:09.936964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:09.936934 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:09.944894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:09.944864 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:05:10.631178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:10.631140 6001 scope.go:122] "RemoveContainer" containerID="e734d8ee9a663874ec2bfe9951f9607e5e896701a4bbde157b783116c9edca7e" Nov 03 17:05:14.039141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:14.039105 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf"] Nov 03 17:05:14.051316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:14.051290 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-526262a1-k6wvf"] Nov 03 17:05:14.792236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:14.792200 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd2d81a-7dff-47c5-a266-dd72cd6768c7" path="/var/lib/kubelet/pods/efd2d81a-7dff-47c5-a266-dd72cd6768c7/volumes" Nov 03 17:05:15.267637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.267604 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:15.273826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.273807 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.275945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.275918 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.276058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.275983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.276180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.276157 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.276340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.276321 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5g7m\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.292390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.292365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:15.300821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.300797 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-76c8387ff3f3\"" Nov 03 17:05:15.301265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.301244 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-df4c99ec3ce4\"" Nov 03 17:05:15.301417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.301296 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ca9d7c\"" Nov 03 17:05:15.301417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.301340 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-e2e--d0a79461-dockercfg-c9d8c\"" Nov 03 17:05:15.301417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.301347 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"openshift-service-ca.crt\"" Nov 03 17:05:15.303488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.303465 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-4ce9c9bbf166\"" Nov 03 17:05:15.303628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.303507 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"routing-overlay\"" Nov 03 17:05:15.303628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.303505 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"kube-root-ca.crt\"" Nov 03 17:05:15.378021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.377996 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5g7m\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.378170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.378078 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.378170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.378128 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.378291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.378272 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.378913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.378889 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.379011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.378961 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.380862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.380841 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.402276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.402253 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5g7m\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:15.588871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:15.588787 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:16.141128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:16.139260 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:17.124017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:17.123980 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" event={"ID":"d3790a1c-6280-4410-860c-81cb47123d68","Type":"ContainerStarted","Data":"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93"} Nov 03 17:05:17.124480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:17.124035 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" event={"ID":"d3790a1c-6280-4410-860c-81cb47123d68","Type":"ContainerStarted","Data":"499b269ae717c524ac5345af8ac3fbaa4ac405062902274c19130a95207b6187"} Nov 03 17:05:17.124480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:17.124106 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:17.172874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:17.172821 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" podStartSLOduration=2.172807386 podStartE2EDuration="2.172807386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:17.169695067 +0000 UTC m=+3012.502908638" watchObservedRunningTime="2026-11-03 17:05:17.172807386 +0000 UTC m=+3012.506020955" Nov 03 17:05:20.403758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.402157 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:20.409388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.409360 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.423169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.423138 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:20.430784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.430754 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"openshift-service-ca.crt\"" Nov 03 17:05:20.431019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.430754 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"agc-cred-a30b47f422e4\"" Nov 03 17:05:20.431019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.430754 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d6fcae\"" Nov 03 17:05:20.431203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.430820 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-e2e--e145110a-dockercfg-jbbzl\"" Nov 03 17:05:20.431203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.430827 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"routing-overlay\"" Nov 03 17:05:20.437546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.437524 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"kube-root-ca.crt\"" Nov 03 17:05:20.446316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.446282 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9sjl\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.446605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.446569 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.446754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.446737 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.446959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.446935 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.556470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.550029 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.556470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.550155 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.556470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.550294 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.581469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.560755 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.581469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.579355 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.591410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.550410 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9sjl\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.656524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.635080 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9sjl\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.709938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.699069 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.734401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.734366 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:20.839199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.839163 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj"] Nov 03 17:05:20.846024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.845989 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:20.869252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.867524 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:05:20.869953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.868426 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:05:20.869953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.868585 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:05:20.869953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.869162 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:05:20.903684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.903578 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj"] Nov 03 17:05:20.915980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.915096 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcxc\" (UniqueName: \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:20.915980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.915179 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:20.915980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:20.915290 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.017182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.017143 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcxc\" (UniqueName: \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.017404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.017221 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.017404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.017307 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.017966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.017940 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.019846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.019820 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.041921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.041889 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcxc\" (UniqueName: \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\") pod \"maas-api-revoke-sub-6232b9b7-lggrj\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.106149 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:21.106117 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387e08b5_48d0_42be_8bf8_228d55ff0db1.slice/crio-b18ed87338be27976876447638db1fae01547229d43fbf965ff2fef50390be9e WatchSource:0}: Error finding container b18ed87338be27976876447638db1fae01547229d43fbf965ff2fef50390be9e: Status 404 returned error can't find the container with id b18ed87338be27976876447638db1fae01547229d43fbf965ff2fef50390be9e Nov 03 17:05:21.106639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.106606 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:21.187492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.187464 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:21.293083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.292658 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" event={"ID":"387e08b5-48d0-42be-8bf8-228d55ff0db1","Type":"ContainerStarted","Data":"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8"} Nov 03 17:05:21.293083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.292745 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:21.293083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.292774 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" event={"ID":"387e08b5-48d0-42be-8bf8-228d55ff0db1","Type":"ContainerStarted","Data":"b18ed87338be27976876447638db1fae01547229d43fbf965ff2fef50390be9e"} Nov 03 17:05:21.340406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.340330 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" podStartSLOduration=1.34030962 podStartE2EDuration="1.34030962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:21.338642406 +0000 UTC m=+3016.671855976" watchObservedRunningTime="2026-11-03 17:05:21.34030962 +0000 UTC m=+3016.673523190" Nov 03 17:05:21.505873 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:21.505841 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c523f1_ae7e_4ef4_970b_8b11f2575511.slice/crio-7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810 WatchSource:0}: Error finding container 7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810: Status 404 returned error can't find the container with id 7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810 Nov 03 17:05:21.506275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:21.506129 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj"] Nov 03 17:05:22.260810 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.260764 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:05:22.304359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:22.304320 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" event={"ID":"c2c523f1-ae7e-4ef4-970b-8b11f2575511","Type":"ContainerStarted","Data":"c26c56663c6e5ab6c4f99a5014cd8b50e0d69dbf67dec30acf33a3d6f388f1b0"} Nov 03 17:05:22.304551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:22.304382 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" event={"ID":"c2c523f1-ae7e-4ef4-970b-8b11f2575511","Type":"ContainerStarted","Data":"7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810"} Nov 03 17:05:22.349902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:22.349844 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" podStartSLOduration=2.349827854 podStartE2EDuration="2.349827854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:22.346740683 +0000 UTC m=+3017.679954243" watchObservedRunningTime="2026-11-03 17:05:22.349827854 +0000 UTC m=+3017.683041423" Nov 03 17:05:22.351601 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.351572 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d6fcae/agc-cred-a30b47f422e4: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:22.351601 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.351598 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:22.351830 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.351682 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials podName:387e08b5-48d0-42be-8bf8-228d55ff0db1 nodeName:}" failed. No retries permitted until 2026-11-03 17:05:22.851661869 +0000 UTC m=+3018.184875447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1") : secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:22.860279 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.860236 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d6fcae/agc-cred-a30b47f422e4: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:22.860279 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.860267 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:22.860774 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:22.860337 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials podName:387e08b5-48d0-42be-8bf8-228d55ff0db1 nodeName:}" failed. No retries permitted until 2026-11-03 17:05:23.860321579 +0000 UTC m=+3019.193535130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1") : secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:23.078471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:23.077505 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k"] Nov 03 17:05:23.116474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:23.096045 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dff75263-skr8k"] Nov 03 17:05:23.386221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:23.385554 6001 generic.go:356] "Generic (PLEG): container finished" podID="c2c523f1-ae7e-4ef4-970b-8b11f2575511" containerID="c26c56663c6e5ab6c4f99a5014cd8b50e0d69dbf67dec30acf33a3d6f388f1b0" exitCode=0 Nov 03 17:05:23.386221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:23.385963 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" event={"ID":"c2c523f1-ae7e-4ef4-970b-8b11f2575511","Type":"ContainerDied","Data":"c26c56663c6e5ab6c4f99a5014cd8b50e0d69dbf67dec30acf33a3d6f388f1b0"} Nov 03 17:05:23.932601 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:23.932563 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d6fcae/agc-cred-a30b47f422e4: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:23.932601 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:23.932592 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:23.933131 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:23.932657 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials podName:387e08b5-48d0-42be-8bf8-228d55ff0db1 nodeName:}" failed. No retries permitted until 2026-11-03 17:05:25.932636782 +0000 UTC m=+3021.265850343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1") : secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:24.162078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.162040 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:05:24.167244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.167219 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.184814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.184714 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-bafaaa823370\"" Nov 03 17:05:24.187213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.187182 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:05:24.237830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.237795 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.238062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.238046 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfj6t\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.238107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.238073 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.238107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.238091 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.339893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.339858 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfj6t\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.339893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.339900 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.340148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.339925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.340148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.340071 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.340729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.340702 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.340865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.340745 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.342807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.342783 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.364347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.364311 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfj6t\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.483420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.483373 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:24.587548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.583965 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:24.664130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.664095 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:24.746976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.746938 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\" (UniqueName: \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\") pod \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " Nov 03 17:05:24.747326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.747309 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\") pod \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " Nov 03 17:05:24.747424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.747367 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\") pod \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\" (UID: \"c2c523f1-ae7e-4ef4-970b-8b11f2575511\") " Nov 03 17:05:24.747685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.747663 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca" pod "c2c523f1-ae7e-4ef4-970b-8b11f2575511" (UID: "c2c523f1-ae7e-4ef4-970b-8b11f2575511"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:24.748065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.748049 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:24.749063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.749038 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc" pod "c2c523f1-ae7e-4ef4-970b-8b11f2575511" (UID: "c2c523f1-ae7e-4ef4-970b-8b11f2575511"). InnerVolumeSpecName "kube-api-access-pwcxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:24.749330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.749314 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca" pod "c2c523f1-ae7e-4ef4-970b-8b11f2575511" (UID: "c2c523f1-ae7e-4ef4-970b-8b11f2575511"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:05:24.792423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.792398 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1" path="/var/lib/kubelet/pods/8fc14c42-bde7-4e2b-8da8-b6f5eb2c3bb1/volumes" Nov 03 17:05:24.820944 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:24.820913 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c40fb4c_5ffb_4239_9a08_ece4522920e9.slice/crio-6e88277e0a3c81897ef37115879eb6716230fdde17c4fe72b2ecb6cdcc17bfb5 WatchSource:0}: Error finding container 6e88277e0a3c81897ef37115879eb6716230fdde17c4fe72b2ecb6cdcc17bfb5: Status 404 returned error can't find the container with id 6e88277e0a3c81897ef37115879eb6716230fdde17c4fe72b2ecb6cdcc17bfb5 Nov 03 17:05:24.821854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.821828 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:05:24.850031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.850003 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c523f1-ae7e-4ef4-970b-8b11f2575511-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:24.850031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:24.850031 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwcxc\" (UniqueName: \"kubernetes.io/projected/c2c523f1-ae7e-4ef4-970b-8b11f2575511-kube-api-access-pwcxc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:25.406970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.406931 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" event={"ID":"5c40fb4c-5ffb-4239-9a08-ece4522920e9","Type":"ContainerStarted","Data":"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f"} Nov 03 17:05:25.407497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.407025 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" event={"ID":"5c40fb4c-5ffb-4239-9a08-ece4522920e9","Type":"ContainerStarted","Data":"6e88277e0a3c81897ef37115879eb6716230fdde17c4fe72b2ecb6cdcc17bfb5"} Nov 03 17:05:25.407497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.407151 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:25.408837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.408810 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" Nov 03 17:05:25.408936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.408839 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj" event={"ID":"c2c523f1-ae7e-4ef4-970b-8b11f2575511","Type":"ContainerDied","Data":"7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810"} Nov 03 17:05:25.408936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.408870 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d81718731b79a811085c05f35f3157f036474a75ae6ce37d9579418d143e810" Nov 03 17:05:25.454460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:25.454398 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" podStartSLOduration=1.454384654 podStartE2EDuration="1.454384654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:25.45097905 +0000 UTC m=+3020.784192619" watchObservedRunningTime="2026-11-03 17:05:25.454384654 +0000 UTC m=+3020.787598239" Nov 03 17:05:26.023401 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:26.023365 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d6fcae/agc-cred-a30b47f422e4: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:26.023401 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:26.023395 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:26.023709 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:26.023474 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials podName:387e08b5-48d0-42be-8bf8-228d55ff0db1 nodeName:}" failed. No retries permitted until 2026-11-03 17:05:30.02345454 +0000 UTC m=+3025.356668103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1") : secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:26.036783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:26.036745 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd"] Nov 03 17:05:26.041090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:26.041057 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27b78354-6f8hd"] Nov 03 17:05:26.170055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:26.170017 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 17:05:26.170301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:26.170275 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" containerID="cri-o://d5e997cec03b80308e5189ab86bcc2af805193ce1bc8a3566e407503437737cd" gracePeriod=30 Nov 03 17:05:26.792555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:26.792519 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed72ab3-ee6a-43f9-a671-4206ef791512" path="/var/lib/kubelet/pods/2ed72ab3-ee6a-43f9-a671-4206ef791512/volumes" Nov 03 17:05:28.027920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.027883 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:28.031941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.031904 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:28.084007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.083974 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:05:28.085013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.084985 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c2c523f1-ae7e-4ef4-970b-8b11f2575511" containerName="revoke-keys" Nov 03 17:05:28.085013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.085010 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c523f1-ae7e-4ef4-970b-8b11f2575511" containerName="revoke-keys" Nov 03 17:05:28.085313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.085296 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c523f1-ae7e-4ef4-970b-8b11f2575511" containerName="revoke-keys" Nov 03 17:05:28.089896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.089868 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.107945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.107915 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-941b3c888a83\"" Nov 03 17:05:28.112729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.112702 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:05:28.187354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.184834 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.189782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.189751 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.190501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.190403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.190732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.190710 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7w9\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.292812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.292717 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.292812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.292799 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7w9\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.293047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.292884 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.293047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.292919 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.293630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.293605 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.293740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.293626 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.295344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.295326 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.317032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.317009 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7w9\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.406297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.406258 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:28.713777 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:28.713735 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2f4b92_79b6_4b2e_9970_cdd06fa95ebc.slice/crio-9442da749a1145b9bf34eb5ce1e67f87aa39aab7a4dc7328329067975ed8288c WatchSource:0}: Error finding container 9442da749a1145b9bf34eb5ce1e67f87aa39aab7a4dc7328329067975ed8288c: Status 404 returned error can't find the container with id 9442da749a1145b9bf34eb5ce1e67f87aa39aab7a4dc7328329067975ed8288c Nov 03 17:05:28.715029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:28.715004 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:05:29.479365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:29.479332 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" event={"ID":"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc","Type":"ContainerStarted","Data":"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b"} Nov 03 17:05:29.479938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:29.479411 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" event={"ID":"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc","Type":"ContainerStarted","Data":"9442da749a1145b9bf34eb5ce1e67f87aa39aab7a4dc7328329067975ed8288c"} Nov 03 17:05:29.479938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:29.479485 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:29.527424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:29.527355 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" podStartSLOduration=1.52733697 podStartE2EDuration="1.52733697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:29.524319021 +0000 UTC m=+3024.857532590" watchObservedRunningTime="2026-11-03 17:05:29.52733697 +0000 UTC m=+3024.860550541" Nov 03 17:05:29.664414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:29.664345 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" podUID="387e08b5-48d0-42be-8bf8-228d55ff0db1" containerName="payload-processing" containerID="cri-o://c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8" gracePeriod=25 Nov 03 17:05:30.005474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.005409 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:30.036677 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:30.036572 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-d6fcae/agc-cred-a30b47f422e4: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:30.036677 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:30.036598 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd: secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:30.036677 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:30.036654 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials podName:387e08b5-48d0-42be-8bf8-228d55ff0db1 nodeName:}" failed. No retries permitted until 2026-11-03 17:05:38.03663647 +0000 UTC m=+3033.369850030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1") : secret "agc-cred-a30b47f422e4" not found Nov 03 17:05:30.137210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137177 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\") pod \"387e08b5-48d0-42be-8bf8-228d55ff0db1\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " Nov 03 17:05:30.137384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137317 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\") pod \"387e08b5-48d0-42be-8bf8-228d55ff0db1\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " Nov 03 17:05:30.137384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137376 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\") pod \"387e08b5-48d0-42be-8bf8-228d55ff0db1\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " Nov 03 17:05:30.137571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137554 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\") pod \"387e08b5-48d0-42be-8bf8-228d55ff0db1\" (UID: \"387e08b5-48d0-42be-8bf8-228d55ff0db1\") " Nov 03 17:05:30.137766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137734 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume" pod "387e08b5-48d0-42be-8bf8-228d55ff0db1" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:30.137958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.137931 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume" pod "387e08b5-48d0-42be-8bf8-228d55ff0db1" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:30.138265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.138247 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:30.138340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.138267 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/387e08b5-48d0-42be-8bf8-228d55ff0db1-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:30.139674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.139648 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials" pod "387e08b5-48d0-42be-8bf8-228d55ff0db1" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:30.139799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.139751 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl" pod "387e08b5-48d0-42be-8bf8-228d55ff0db1" (UID: "387e08b5-48d0-42be-8bf8-228d55ff0db1"). InnerVolumeSpecName "kube-api-access-f9sjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:30.239645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.239612 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9sjl\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-kube-api-access-f9sjl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:30.239645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.239641 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/387e08b5-48d0-42be-8bf8-228d55ff0db1-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:30.489501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.489467 6001 generic.go:356] "Generic (PLEG): container finished" podID="387e08b5-48d0-42be-8bf8-228d55ff0db1" containerID="c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8" exitCode=0 Nov 03 17:05:30.489932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.489513 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" Nov 03 17:05:30.489932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.489552 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" event={"ID":"387e08b5-48d0-42be-8bf8-228d55ff0db1","Type":"ContainerDied","Data":"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8"} Nov 03 17:05:30.489932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.489610 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd" event={"ID":"387e08b5-48d0-42be-8bf8-228d55ff0db1","Type":"ContainerDied","Data":"b18ed87338be27976876447638db1fae01547229d43fbf965ff2fef50390be9e"} Nov 03 17:05:30.489932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.489640 6001 scope.go:122] "RemoveContainer" containerID="c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8" Nov 03 17:05:30.501570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.501551 6001 scope.go:122] "RemoveContainer" containerID="c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8" Nov 03 17:05:30.501866 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:30.501843 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8\": container with ID starting with c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8 not found: ID does not exist" containerID="c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8" Nov 03 17:05:30.501916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.501874 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8"} err="failed to get container status \"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8\": rpc error: code = NotFound desc = could not find container \"c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8\": container with ID starting with c8bdec031eeb07216face2b65501aff1dd1251883fb1592490b0ab1ab47d5de8 not found: ID does not exist" Nov 03 17:05:30.551786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.551753 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:30.563984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.563938 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5455fqknd"] Nov 03 17:05:30.792670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:30.792586 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387e08b5-48d0-42be-8bf8-228d55ff0db1" path="/var/lib/kubelet/pods/387e08b5-48d0-42be-8bf8-228d55ff0db1/volumes" Nov 03 17:05:31.503235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.503197 6001 generic.go:356] "Generic (PLEG): container finished" podID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerID="d5e997cec03b80308e5189ab86bcc2af805193ce1bc8a3566e407503437737cd" exitCode=0 Nov 03 17:05:31.503688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.503271 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" event={"ID":"964b9fa7-e304-4c95-aa7b-c152c097da64","Type":"ContainerDied","Data":"d5e997cec03b80308e5189ab86bcc2af805193ce1bc8a3566e407503437737cd"} Nov 03 17:05:31.552297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.552268 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 17:05:31.677992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.677949 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678102 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678193 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678237 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678338 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678367 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678420 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:31.678490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678467 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678510 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\") pod \"964b9fa7-e304-4c95-aa7b-c152c097da64\" (UID: \"964b9fa7-e304-4c95-aa7b-c152c097da64\") " Nov 03 17:05:31.678794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678683 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:31.678998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678845 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:31.678998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.678868 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:31.679145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679071 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:31.679586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679566 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.679698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679590 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.679698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679605 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964b9fa7-e304-4c95-aa7b-c152c097da64-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.679698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679618 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.679698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.679631 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.681735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.681701 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:05:31.682665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.682523 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:31.682665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.682632 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "kube-api-access-vzzkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:31.684349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.684315 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token" pod "964b9fa7-e304-4c95-aa7b-c152c097da64" (UID: "964b9fa7-e304-4c95-aa7b-c152c097da64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:31.781461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.781326 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzkv\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-kube-api-access-vzzkv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.781461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.781359 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.781461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.781368 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:31.781461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:31.781377 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964b9fa7-e304-4c95-aa7b-c152c097da64-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:32.513859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.513829 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" Nov 03 17:05:32.514380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.513857 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv" event={"ID":"964b9fa7-e304-4c95-aa7b-c152c097da64","Type":"ContainerDied","Data":"68da4eb3dd5be1ea465fb38b6578c5740ec156582c7b0a953122fa2b478bd595"} Nov 03 17:05:32.514380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.513926 6001 scope.go:122] "RemoveContainer" containerID="d5e997cec03b80308e5189ab86bcc2af805193ce1bc8a3566e407503437737cd" Nov 03 17:05:32.635407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.635335 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 17:05:32.659726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.659688 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-mc5cv"] Nov 03 17:05:32.814250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:32.814149 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" path="/var/lib/kubelet/pods/964b9fa7-e304-4c95-aa7b-c152c097da64/volumes" Nov 03 17:05:33.028489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.028401 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" podUID="d3790a1c-6280-4410-860c-81cb47123d68" containerName="payload-processing" containerID="cri-o://61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93" gracePeriod=25 Nov 03 17:05:33.390548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.390516 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:33.421612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.421501 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\") pod \"d3790a1c-6280-4410-860c-81cb47123d68\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " Nov 03 17:05:33.421806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.421720 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\") pod \"d3790a1c-6280-4410-860c-81cb47123d68\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " Nov 03 17:05:33.421893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.421814 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\") pod \"d3790a1c-6280-4410-860c-81cb47123d68\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " Nov 03 17:05:33.422124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.422075 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\") pod \"d3790a1c-6280-4410-860c-81cb47123d68\" (UID: \"d3790a1c-6280-4410-860c-81cb47123d68\") " Nov 03 17:05:33.422492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.422424 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume" pod "d3790a1c-6280-4410-860c-81cb47123d68" (UID: "d3790a1c-6280-4410-860c-81cb47123d68"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:33.422654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.422617 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume" pod "d3790a1c-6280-4410-860c-81cb47123d68" (UID: "d3790a1c-6280-4410-860c-81cb47123d68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:33.423846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.423813 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:33.423965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.423848 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d3790a1c-6280-4410-860c-81cb47123d68-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:33.424945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.424919 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m" pod "d3790a1c-6280-4410-860c-81cb47123d68" (UID: "d3790a1c-6280-4410-860c-81cb47123d68"). InnerVolumeSpecName "kube-api-access-s5g7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:33.425265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.425244 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials" pod "d3790a1c-6280-4410-860c-81cb47123d68" (UID: "d3790a1c-6280-4410-860c-81cb47123d68"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:33.525506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.525467 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5g7m\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-kube-api-access-s5g7m\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:33.525506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.525498 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d3790a1c-6280-4410-860c-81cb47123d68-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:33.527780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.527745 6001 generic.go:356] "Generic (PLEG): container finished" podID="d3790a1c-6280-4410-860c-81cb47123d68" containerID="61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93" exitCode=0 Nov 03 17:05:33.527963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.527807 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" Nov 03 17:05:33.527963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.527837 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" event={"ID":"d3790a1c-6280-4410-860c-81cb47123d68","Type":"ContainerDied","Data":"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93"} Nov 03 17:05:33.527963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.527909 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn" event={"ID":"d3790a1c-6280-4410-860c-81cb47123d68","Type":"ContainerDied","Data":"499b269ae717c524ac5345af8ac3fbaa4ac405062902274c19130a95207b6187"} Nov 03 17:05:33.527963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.527930 6001 scope.go:122] "RemoveContainer" containerID="61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93" Nov 03 17:05:33.557937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.557913 6001 scope.go:122] "RemoveContainer" containerID="61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93" Nov 03 17:05:33.558348 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:33.558322 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93\": container with ID starting with 61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93 not found: ID does not exist" containerID="61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93" Nov 03 17:05:33.558506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.558361 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93"} err="failed to get container status \"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93\": rpc error: code = NotFound desc = could not find container \"61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93\": container with ID starting with 61aa0e12036b14b9dd5a22f7f9af3128943e6c734d08ebc1082fab848a1ddf93 not found: ID does not exist" Nov 03 17:05:33.588162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.588119 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:33.594183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:33.594149 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-658bx9zxn"] Nov 03 17:05:34.962472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:34.961894 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3790a1c-6280-4410-860c-81cb47123d68" path="/var/lib/kubelet/pods/d3790a1c-6280-4410-860c-81cb47123d68/volumes" Nov 03 17:05:35.989166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.989131 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:05:35.990240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990213 6001 cpu_manager.go:414] "containerMap: removing container" podUID="387e08b5-48d0-42be-8bf8-228d55ff0db1" containerName="payload-processing" Nov 03 17:05:35.990406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990241 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387e08b5-48d0-42be-8bf8-228d55ff0db1" containerName="payload-processing" Nov 03 17:05:35.990406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990278 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d3790a1c-6280-4410-860c-81cb47123d68" containerName="payload-processing" Nov 03 17:05:35.990406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990286 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3790a1c-6280-4410-860c-81cb47123d68" containerName="payload-processing" Nov 03 17:05:35.990406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990345 6001 cpu_manager.go:414] "containerMap: removing container" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" Nov 03 17:05:35.990406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990353 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" Nov 03 17:05:35.990707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990602 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="387e08b5-48d0-42be-8bf8-228d55ff0db1" containerName="payload-processing" Nov 03 17:05:35.990707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990631 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3790a1c-6280-4410-860c-81cb47123d68" containerName="payload-processing" Nov 03 17:05:35.990707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.990657 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="964b9fa7-e304-4c95-aa7b-c152c097da64" containerName="istio-proxy" Nov 03 17:05:35.994600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:35.994578 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.012602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.012570 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e0e63d-openshift-default-dockercfg-72z97\"" Nov 03 17:05:36.018496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.018467 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:05:36.052584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052545 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.052807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052613 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.052807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052798 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.052947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052843 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl588\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.052947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052941 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.053064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.052968 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.053125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.053066 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.053125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.053098 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.053250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.053135 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154192 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154243 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154270 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154388 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154439 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154573 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154606 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl588\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154692 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154710 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.154985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.154911 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.155136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.155105 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.155205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.155136 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.155715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.155689 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.157092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.157065 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.157205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.157136 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.178866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.178837 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.179031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.179009 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl588\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.314709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.314614 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:36.450266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.450224 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:05:36.533031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.532984 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:05:36.660713 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:36.660672 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64399d01_9590_4dee_a4ac_5077a4acfc70.slice/crio-cf05fa4c3df4b36c5186042c801fd8f22afabf455e17fc208469a64eddef33c6 WatchSource:0}: Error finding container cf05fa4c3df4b36c5186042c801fd8f22afabf455e17fc208469a64eddef33c6: Status 404 returned error can't find the container with id cf05fa4c3df4b36c5186042c801fd8f22afabf455e17fc208469a64eddef33c6 Nov 03 17:05:36.661267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.661241 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:05:36.663791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.663757 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:36.663872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.663823 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:36.663872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:36.663854 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:37.582707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:37.582671 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" event={"ID":"64399d01-9590-4dee-a4ac-5077a4acfc70","Type":"ContainerStarted","Data":"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3"} Nov 03 17:05:37.583114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:37.582732 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" event={"ID":"64399d01-9590-4dee-a4ac-5077a4acfc70","Type":"ContainerStarted","Data":"cf05fa4c3df4b36c5186042c801fd8f22afabf455e17fc208469a64eddef33c6"} Nov 03 17:05:37.675095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:37.675029 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podStartSLOduration=2.6750075239999997 podStartE2EDuration="2.675007524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:37.669232097 +0000 UTC m=+3033.002445671" watchObservedRunningTime="2026-11-03 17:05:37.675007524 +0000 UTC m=+3033.008221094" Nov 03 17:05:38.314888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:38.314852 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:38.316815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:38.316786 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 03 17:05:38.316963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:38.316840 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 03 17:05:39.315422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:39.315386 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 03 17:05:39.315961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:39.315474 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 03 17:05:40.315336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:40.315302 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 03 17:05:40.315554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:40.315370 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 03 17:05:40.493551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:40.493521 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:05:41.315638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.315585 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 03 17:05:41.316078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.315661 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 03 17:05:41.533721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.533673 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" podUID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" containerName="payload-processing" containerID="cri-o://5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0" gracePeriod=25 Nov 03 17:05:41.703106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.703066 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:05:41.708971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.708940 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:05:41.709156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.709054 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806033 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsfkd\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806195 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806227 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806293 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806321 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806370 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806553 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806657 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.808470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.806764 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919225 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919650 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919618 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsfkd\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919787 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919881 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.919943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919911 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.920129 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.920232 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.920494 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.919614 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.920790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.920768 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.921239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.921218 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.921409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.921392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.926907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.926874 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.927970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.927940 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.967470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.967280 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsfkd\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:41.978032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:41.977993 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:42.027920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.027893 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:42.034655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.034628 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:05:42.229184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.229127 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\") pod \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " Nov 03 17:05:42.229413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.229200 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\") pod \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " Nov 03 17:05:42.229413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.229279 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\") pod \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " Nov 03 17:05:42.229557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.229485 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\") pod \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\" (UID: \"b462b489-4dd9-4c29-9ba9-cc60c24c1d19\") " Nov 03 17:05:42.229880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.229837 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume" pod "b462b489-4dd9-4c29-9ba9-cc60c24c1d19" (UID: "b462b489-4dd9-4c29-9ba9-cc60c24c1d19"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:42.230258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.230233 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume" pod "b462b489-4dd9-4c29-9ba9-cc60c24c1d19" (UID: "b462b489-4dd9-4c29-9ba9-cc60c24c1d19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:42.230378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.230361 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:42.234544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.234517 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6" pod "b462b489-4dd9-4c29-9ba9-cc60c24c1d19" (UID: "b462b489-4dd9-4c29-9ba9-cc60c24c1d19"). InnerVolumeSpecName "kube-api-access-wd9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:42.235170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.235146 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials" pod "b462b489-4dd9-4c29-9ba9-cc60c24c1d19" (UID: "b462b489-4dd9-4c29-9ba9-cc60c24c1d19"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:42.315439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.315403 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body= Nov 03 17:05:42.315634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.315481 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" Nov 03 17:05:42.332084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.332051 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:42.332084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.332080 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:42.332084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.332091 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd9c6\" (UniqueName: \"kubernetes.io/projected/b462b489-4dd9-4c29-9ba9-cc60c24c1d19-kube-api-access-wd9c6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:05:42.398150 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:05:42.398116 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a5f7f1_2908_432e_9c29_dd221c55217c.slice/crio-98ce266b49d6be09c40d6bd04ddc49fba518010124630019a6ea87affb872caa WatchSource:0}: Error finding container 98ce266b49d6be09c40d6bd04ddc49fba518010124630019a6ea87affb872caa: Status 404 returned error can't find the container with id 98ce266b49d6be09c40d6bd04ddc49fba518010124630019a6ea87affb872caa Nov 03 17:05:42.399847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.399825 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:05:42.401146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.401110 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:42.401255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.401192 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:42.401255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.401236 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:05:42.705988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.705952 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" event={"ID":"b2a5f7f1-2908-432e-9c29-dd221c55217c","Type":"ContainerStarted","Data":"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f"} Nov 03 17:05:42.706183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.706017 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" event={"ID":"b2a5f7f1-2908-432e-9c29-dd221c55217c","Type":"ContainerStarted","Data":"98ce266b49d6be09c40d6bd04ddc49fba518010124630019a6ea87affb872caa"} Nov 03 17:05:42.708077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.708051 6001 generic.go:356] "Generic (PLEG): container finished" podID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" containerID="5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0" exitCode=0 Nov 03 17:05:42.708233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.708095 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" Nov 03 17:05:42.708233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.708133 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" event={"ID":"b462b489-4dd9-4c29-9ba9-cc60c24c1d19","Type":"ContainerDied","Data":"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0"} Nov 03 17:05:42.708233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.708180 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7" event={"ID":"b462b489-4dd9-4c29-9ba9-cc60c24c1d19","Type":"ContainerDied","Data":"554a165d39477855b8e53bde37279108d29f019f0850cf5560dec9d11cb7987e"} Nov 03 17:05:42.708233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.708203 6001 scope.go:122] "RemoveContainer" containerID="5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0" Nov 03 17:05:42.720838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.720814 6001 scope.go:122] "RemoveContainer" containerID="5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0" Nov 03 17:05:42.721213 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:42.721186 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0\": container with ID starting with 5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0 not found: ID does not exist" containerID="5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0" Nov 03 17:05:42.721261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.721227 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0"} err="failed to get container status \"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0\": rpc error: code = NotFound desc = could not find container \"5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0\": container with ID starting with 5c977dde328e829a99f97642b19f209e4d5f35fcaec6096692bf5ea2832139c0 not found: ID does not exist" Nov 03 17:05:42.773555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.770243 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podStartSLOduration=1.770221024 podStartE2EDuration="1.770221024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:42.767756175 +0000 UTC m=+3038.100969744" watchObservedRunningTime="2026-11-03 17:05:42.770221024 +0000 UTC m=+3038.103434595" Nov 03 17:05:42.811189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.811153 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:05:42.815301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:42.815271 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78487wpd7"] Nov 03 17:05:43.029389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.029286 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:43.031154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.031114 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:43.031312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.031185 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:43.320966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.320875 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:43.321257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.321234 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:43.321930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:43.321904 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:05:44.029585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:44.029508 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:44.030025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:44.029580 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:44.793222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:44.793184 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" path="/var/lib/kubelet/pods/b462b489-4dd9-4c29-9ba9-cc60c24c1d19/volumes" Nov 03 17:05:45.029243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:45.029205 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:45.029454 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:45.029272 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:46.039469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:46.032129 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:46.039469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:46.032233 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:47.028999 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:47.028904 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:47.030083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:47.030044 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:48.029351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:48.029318 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:48.029789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:48.029390 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:49.029361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:49.029330 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" start-of-body= Nov 03 17:05:49.029791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:49.029413 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.29:15021/healthz/ready\": dial tcp 10.134.1.29:15021: connect: connection refused" Nov 03 17:05:50.034127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:50.034089 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:50.034610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:50.034384 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:50.035291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:05:50.035269 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:05:52.255058 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:05:52.255016 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:06:10.344632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:10.344579 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:06:10.356613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:10.356560 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:06:10.444469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:10.443546 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 17:06:10.687534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:10.684848 6001 scope.go:122] "RemoveContainer" containerID="283af6d8fb7cc764030243f7c929680122607bd380db079e577f78f20e7adc6f" Nov 03 17:06:22.255666 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:06:22.255576 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:06:38.670355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.670313 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6"] Nov 03 17:06:38.671361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.671340 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" containerName="payload-processing" Nov 03 17:06:38.671409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.671364 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" containerName="payload-processing" Nov 03 17:06:38.671612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.671599 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b462b489-4dd9-4c29-9ba9-cc60c24c1d19" containerName="payload-processing" Nov 03 17:06:38.675654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.675636 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.694475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.694431 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6"] Nov 03 17:06:38.705792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.705758 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:06:38.706005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.705768 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:06:38.706005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.705823 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:06:38.706106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.705926 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:06:38.713202 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.713173 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dtr6\" (UniqueName: \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.713543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.713519 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.713693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.713564 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.816188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.816146 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.816396 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.816221 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.816524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.816398 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dtr6\" (UniqueName: \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.817007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.816981 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.818633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.818606 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.841018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.840990 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dtr6\" (UniqueName: \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\") pod \"maas-api-revoke-sub-96f09a19-h8nf6\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:38.993322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:38.993286 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:39.298158 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:06:39.298118 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6009a3_2284_47ba_8c9a_3fb78c53e2e0.slice/crio-11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97 WatchSource:0}: Error finding container 11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97: Status 404 returned error can't find the container with id 11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97 Nov 03 17:06:39.299733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:39.299702 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6"] Nov 03 17:06:39.676515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:39.676479 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" event={"ID":"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0","Type":"ContainerStarted","Data":"9b99b1ad98cb051f74c8d4accbccbe1a78c5599f16b689f6299004fce5043c37"} Nov 03 17:06:39.676936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:39.676533 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" event={"ID":"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0","Type":"ContainerStarted","Data":"11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97"} Nov 03 17:06:39.728169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:39.728110 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" podStartSLOduration=1.728096498 podStartE2EDuration="1.728096498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:39.726961224 +0000 UTC m=+3095.060174792" watchObservedRunningTime="2026-11-03 17:06:39.728096498 +0000 UTC m=+3095.061310067" Nov 03 17:06:40.686950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:40.686856 6001 generic.go:356] "Generic (PLEG): container finished" podID="0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" containerID="9b99b1ad98cb051f74c8d4accbccbe1a78c5599f16b689f6299004fce5043c37" exitCode=0 Nov 03 17:06:40.687436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:40.686974 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" event={"ID":"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0","Type":"ContainerDied","Data":"9b99b1ad98cb051f74c8d4accbccbe1a78c5599f16b689f6299004fce5043c37"} Nov 03 17:06:41.882354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.882319 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:41.976557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.976515 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\" (UniqueName: \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\") pod \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " Nov 03 17:06:41.976853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.976830 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\") pod \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " Nov 03 17:06:41.977014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.976993 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\") pod \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\" (UID: \"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0\") " Nov 03 17:06:41.977415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.977384 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca" pod "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" (UID: "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:41.977947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.977921 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:41.979238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.979204 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6" pod "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" (UID: "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0"). InnerVolumeSpecName "kube-api-access-6dtr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:41.979357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:41.979330 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca" pod "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" (UID: "0b6009a3-2284-47ba-8c9a-3fb78c53e2e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:06:42.079300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:42.079259 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dtr6\" (UniqueName: \"kubernetes.io/projected/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-kube-api-access-6dtr6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:42.079300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:42.079297 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:42.706290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:42.706243 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" event={"ID":"0b6009a3-2284-47ba-8c9a-3fb78c53e2e0","Type":"ContainerDied","Data":"11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97"} Nov 03 17:06:42.706504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:42.706297 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ac905279d6ea86c26fd9c547ddefeaa36784b2971ffa3a48994232478f8f97" Nov 03 17:06:42.706504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:42.706294 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6" Nov 03 17:06:49.297469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.297425 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6"] Nov 03 17:06:49.298315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.298297 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" containerName="revoke-keys" Nov 03 17:06:49.298367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.298317 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" containerName="revoke-keys" Nov 03 17:06:49.298505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.298495 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" containerName="revoke-keys" Nov 03 17:06:49.302705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.302684 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.312232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.312201 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kbs\" (UniqueName: \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.312398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.312379 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.312618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.312592 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.318745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.318722 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:06:49.322258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.322229 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6"] Nov 03 17:06:49.329557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.329532 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:06:49.329701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.329561 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:06:49.331738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.331720 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:06:49.428874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.428823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kbs\" (UniqueName: \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.454781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.454743 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.466481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.466426 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.471930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.467421 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.473410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.473378 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kbs\" (UniqueName: \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.474718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.474148 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35db2b49-rqfl6\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.619982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.619883 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:49.933129 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:06:49.933098 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f0b10d_ee03_4dfb_9fff_41ba0b4a6103.slice/crio-ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b WatchSource:0}: Error finding container ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b: Status 404 returned error can't find the container with id ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b Nov 03 17:06:49.934069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:49.934029 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6"] Nov 03 17:06:50.817434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:50.817395 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" event={"ID":"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103","Type":"ContainerStarted","Data":"51edd7a28a96242121a2eec0d032a95af5c3145f2d0984124276a60f09a386ca"} Nov 03 17:06:50.817979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:50.817493 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" event={"ID":"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103","Type":"ContainerStarted","Data":"ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b"} Nov 03 17:06:50.864016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:50.863964 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" podStartSLOduration=1.86394948 podStartE2EDuration="1.86394948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:50.862742504 +0000 UTC m=+3106.195956076" watchObservedRunningTime="2026-11-03 17:06:50.86394948 +0000 UTC m=+3106.197163049" Nov 03 17:06:51.828515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:51.828473 6001 generic.go:356] "Generic (PLEG): container finished" podID="d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" containerID="51edd7a28a96242121a2eec0d032a95af5c3145f2d0984124276a60f09a386ca" exitCode=0 Nov 03 17:06:51.829017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:51.828551 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" event={"ID":"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103","Type":"ContainerDied","Data":"51edd7a28a96242121a2eec0d032a95af5c3145f2d0984124276a60f09a386ca"} Nov 03 17:06:52.254714 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:06:52.254675 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:06:53.023356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.023326 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:53.144594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.144500 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\") pod \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " Nov 03 17:06:53.144784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.144669 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\") pod \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " Nov 03 17:06:53.144848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.144834 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\" (UniqueName: \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\") pod \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\" (UID: \"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103\") " Nov 03 17:06:53.144887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.144856 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca" pod "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" (UID: "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:53.145515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.145491 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:53.146954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.146925 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca" pod "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" (UID: "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:06:53.146954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.146945 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs" pod "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" (UID: "d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103"). InnerVolumeSpecName "kube-api-access-r4kbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:53.247658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.247621 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:53.247658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.247652 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4kbs\" (UniqueName: \"kubernetes.io/projected/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103-kube-api-access-r4kbs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:06:53.846952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.846920 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" Nov 03 17:06:53.847142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.846947 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6" event={"ID":"d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103","Type":"ContainerDied","Data":"ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b"} Nov 03 17:06:53.847142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:06:53.847004 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ded20018c2e862ee5580500f7fb9630af0273e4b60753dd13c7c7d160387a67b" Nov 03 17:07:01.763457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.763408 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls"] Nov 03 17:07:01.764344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.764325 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" containerName="revoke-keys" Nov 03 17:07:01.764403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.764345 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" containerName="revoke-keys" Nov 03 17:07:01.764518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.764507 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" containerName="revoke-keys" Nov 03 17:07:01.769421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.769400 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:01.794662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.794615 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:07:01.794859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.794774 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:07:01.795848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.795815 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls"] Nov 03 17:07:01.798827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.798789 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:07:01.798962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.798913 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:07:01.916274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.916234 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgxbc\" (UniqueName: \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:01.916513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.916429 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:01.916710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:01.916682 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.018575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.018478 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.018749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.018706 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgxbc\" (UniqueName: \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.018800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.018746 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.019381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.019353 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.021311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.021286 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.044000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.043961 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgxbc\" (UniqueName: \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\") pod \"maas-api-revoke-sub-d192a6c8-xw5ls\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.112922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.112884 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:02.425326 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:07:02.425291 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06424e2_36e1_4824_a50c_293db164710f.slice/crio-74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b WatchSource:0}: Error finding container 74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b: Status 404 returned error can't find the container with id 74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b Nov 03 17:07:02.426417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.426391 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls"] Nov 03 17:07:02.993850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.993807 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" event={"ID":"e06424e2-36e1-4824-a50c-293db164710f","Type":"ContainerStarted","Data":"00d06d709478e12e69a63c2afe6c627d019cf47eb674e4b159d49378eb1355ff"} Nov 03 17:07:02.994249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:02.993875 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" event={"ID":"e06424e2-36e1-4824-a50c-293db164710f","Type":"ContainerStarted","Data":"74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b"} Nov 03 17:07:03.046085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:03.046027 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" podStartSLOduration=2.04601347 podStartE2EDuration="2.04601347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:03.041931961 +0000 UTC m=+3118.375145554" watchObservedRunningTime="2026-11-03 17:07:03.04601347 +0000 UTC m=+3118.379227039" Nov 03 17:07:04.072468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:04.067746 6001 generic.go:356] "Generic (PLEG): container finished" podID="e06424e2-36e1-4824-a50c-293db164710f" containerID="00d06d709478e12e69a63c2afe6c627d019cf47eb674e4b159d49378eb1355ff" exitCode=0 Nov 03 17:07:04.072468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:04.067956 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" event={"ID":"e06424e2-36e1-4824-a50c-293db164710f","Type":"ContainerDied","Data":"00d06d709478e12e69a63c2afe6c627d019cf47eb674e4b159d49378eb1355ff"} Nov 03 17:07:05.826603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:05.826575 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:06.022353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.022316 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\") pod \"e06424e2-36e1-4824-a50c-293db164710f\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " Nov 03 17:07:06.022590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.022364 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\") pod \"e06424e2-36e1-4824-a50c-293db164710f\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " Nov 03 17:07:06.022590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.022472 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\" (UniqueName: \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\") pod \"e06424e2-36e1-4824-a50c-293db164710f\" (UID: \"e06424e2-36e1-4824-a50c-293db164710f\") " Nov 03 17:07:06.022858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.022832 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca" pod "e06424e2-36e1-4824-a50c-293db164710f" (UID: "e06424e2-36e1-4824-a50c-293db164710f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:06.023172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.023152 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06424e2-36e1-4824-a50c-293db164710f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:06.024672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.024646 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca" pod "e06424e2-36e1-4824-a50c-293db164710f" (UID: "e06424e2-36e1-4824-a50c-293db164710f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:07:06.024843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.024828 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc" pod "e06424e2-36e1-4824-a50c-293db164710f" (UID: "e06424e2-36e1-4824-a50c-293db164710f"). InnerVolumeSpecName "kube-api-access-jgxbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:06.124463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.124417 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06424e2-36e1-4824-a50c-293db164710f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:06.124463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.124467 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgxbc\" (UniqueName: \"kubernetes.io/projected/e06424e2-36e1-4824-a50c-293db164710f-kube-api-access-jgxbc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:06.175172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.175139 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" Nov 03 17:07:06.175346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.175170 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls" event={"ID":"e06424e2-36e1-4824-a50c-293db164710f","Type":"ContainerDied","Data":"74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b"} Nov 03 17:07:06.175346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:06.175233 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74ba6c74903991700c95b398d029a67ff5c977634bafb523c69c342ac46e9f8b" Nov 03 17:07:07.045428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.045390 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r"] Nov 03 17:07:07.046688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.046660 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e06424e2-36e1-4824-a50c-293db164710f" containerName="revoke-keys" Nov 03 17:07:07.046688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.046685 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06424e2-36e1-4824-a50c-293db164710f" containerName="revoke-keys" Nov 03 17:07:07.046955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.046939 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06424e2-36e1-4824-a50c-293db164710f" containerName="revoke-keys" Nov 03 17:07:07.051072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.051047 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.069716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.069683 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r"] Nov 03 17:07:07.077103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.077076 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:07:07.077230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.077129 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:07:07.077230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.077185 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:07:07.079072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.079052 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:07:07.142720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.142688 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.142956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.142730 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.143041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.143024 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtr6c\" (UniqueName: \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.244785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.244752 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtr6c\" (UniqueName: \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.244950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.244816 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.244950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.244840 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.245619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.245596 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.247205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.247184 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.268749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.268723 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtr6c\" (UniqueName: \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\") pod \"maas-api-revoke-sub-f25d2502-gsf6r\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.368791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.368705 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:07.681708 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:07:07.681675 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d3bb9a_723b_418c_a670_4935c13e97d7.slice/crio-b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107 WatchSource:0}: Error finding container b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107: Status 404 returned error can't find the container with id b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107 Nov 03 17:07:07.683556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:07.683506 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r"] Nov 03 17:07:08.194158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:08.194115 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" event={"ID":"f7d3bb9a-723b-418c-a670-4935c13e97d7","Type":"ContainerStarted","Data":"6a984215e753d6819562b7b4679fd34df75ee567f0ff2e755edb7c0287a43d1c"} Nov 03 17:07:08.194585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:08.194181 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" event={"ID":"f7d3bb9a-723b-418c-a670-4935c13e97d7","Type":"ContainerStarted","Data":"b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107"} Nov 03 17:07:08.237548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:08.237491 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" podStartSLOduration=1.237478399 podStartE2EDuration="1.237478399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:08.237071102 +0000 UTC m=+3123.570284670" watchObservedRunningTime="2026-11-03 17:07:08.237478399 +0000 UTC m=+3123.570691968" Nov 03 17:07:09.203662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:09.203622 6001 generic.go:356] "Generic (PLEG): container finished" podID="f7d3bb9a-723b-418c-a670-4935c13e97d7" containerID="6a984215e753d6819562b7b4679fd34df75ee567f0ff2e755edb7c0287a43d1c" exitCode=0 Nov 03 17:07:09.204074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:09.203704 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" event={"ID":"f7d3bb9a-723b-418c-a670-4935c13e97d7","Type":"ContainerDied","Data":"6a984215e753d6819562b7b4679fd34df75ee567f0ff2e755edb7c0287a43d1c"} Nov 03 17:07:10.402300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.402268 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:10.482556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.482519 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\" (UniqueName: \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\") pod \"f7d3bb9a-723b-418c-a670-4935c13e97d7\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " Nov 03 17:07:10.482878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.482863 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\") pod \"f7d3bb9a-723b-418c-a670-4935c13e97d7\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " Nov 03 17:07:10.482937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.482900 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\") pod \"f7d3bb9a-723b-418c-a670-4935c13e97d7\" (UID: \"f7d3bb9a-723b-418c-a670-4935c13e97d7\") " Nov 03 17:07:10.483227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.483201 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca" pod "f7d3bb9a-723b-418c-a670-4935c13e97d7" (UID: "f7d3bb9a-723b-418c-a670-4935c13e97d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:10.483563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.483542 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:10.484740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.484715 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c" pod "f7d3bb9a-723b-418c-a670-4935c13e97d7" (UID: "f7d3bb9a-723b-418c-a670-4935c13e97d7"). InnerVolumeSpecName "kube-api-access-mtr6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:10.485143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.485122 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca" pod "f7d3bb9a-723b-418c-a670-4935c13e97d7" (UID: "f7d3bb9a-723b-418c-a670-4935c13e97d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:07:10.585720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.585691 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7d3bb9a-723b-418c-a670-4935c13e97d7-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:10.585720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.585719 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtr6c\" (UniqueName: \"kubernetes.io/projected/f7d3bb9a-723b-418c-a670-4935c13e97d7-kube-api-access-mtr6c\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:10.946470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:10.942222 6001 scope.go:122] "RemoveContainer" containerID="e0ce659577b5fc538e117c42387f0a1668cfe02b7788a46652500c3ed5f37933" Nov 03 17:07:11.034266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:11.034129 6001 scope.go:122] "RemoveContainer" containerID="d07d750b1dfee2aa81e823a4b11a3de35d35d1ca6eb8291986300e0db512bf1f" Nov 03 17:07:11.070487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:11.070283 6001 scope.go:122] "RemoveContainer" containerID="f61da15aabc4f8c2c513d391ae05305ed193fe11f520ab9ae148d8f37f2092ac" Nov 03 17:07:11.359708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:11.355919 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" Nov 03 17:07:11.359708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:11.356023 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r" event={"ID":"f7d3bb9a-723b-418c-a670-4935c13e97d7","Type":"ContainerDied","Data":"b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107"} Nov 03 17:07:11.359708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:11.356078 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b61532c1c39c6064ca5327d24cbc8df7e58b07053c996f3245a7ee866e3fe107" Nov 03 17:07:18.701243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.701207 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5"] Nov 03 17:07:18.703374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.703345 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f7d3bb9a-723b-418c-a670-4935c13e97d7" containerName="revoke-keys" Nov 03 17:07:18.703560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.703544 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7d3bb9a-723b-418c-a670-4935c13e97d7" containerName="revoke-keys" Nov 03 17:07:18.703890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.703875 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7d3bb9a-723b-418c-a670-4935c13e97d7" containerName="revoke-keys" Nov 03 17:07:18.714473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.710798 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.742054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.742021 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:07:18.742392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.742375 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:07:18.742525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.742092 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:07:18.765833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.765805 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:07:18.801472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.790882 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5"] Nov 03 17:07:18.884478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.864155 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.884478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.864217 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.884478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.864388 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdkf9\" (UniqueName: \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.972485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.972434 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdkf9\" (UniqueName: \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.972767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.972749 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.972846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.972791 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.973424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.973400 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.975281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.975260 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:18.996824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:18.996789 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdkf9\" (UniqueName: \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\") pod \"maas-api-revoke-sub-0ff10a03-ppnr5\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:19.092966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:19.092927 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:19.588840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:19.588757 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5"] Nov 03 17:07:19.593045 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:07:19.593004 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b217f56_85d4_4515_9977_b58bebae9ae4.slice/crio-cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845 WatchSource:0}: Error finding container cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845: Status 404 returned error can't find the container with id cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845 Nov 03 17:07:20.443364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:20.443330 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" event={"ID":"2b217f56-85d4-4515-9977-b58bebae9ae4","Type":"ContainerStarted","Data":"e35eba4d65147c00473df80b06db4e6cfab1d5196b215991dc25d0130403e237"} Nov 03 17:07:20.443903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:20.443387 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" event={"ID":"2b217f56-85d4-4515-9977-b58bebae9ae4","Type":"ContainerStarted","Data":"cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845"} Nov 03 17:07:21.454695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:21.454660 6001 generic.go:356] "Generic (PLEG): container finished" podID="2b217f56-85d4-4515-9977-b58bebae9ae4" containerID="e35eba4d65147c00473df80b06db4e6cfab1d5196b215991dc25d0130403e237" exitCode=0 Nov 03 17:07:21.455109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:21.454743 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" event={"ID":"2b217f56-85d4-4515-9977-b58bebae9ae4","Type":"ContainerDied","Data":"e35eba4d65147c00473df80b06db4e6cfab1d5196b215991dc25d0130403e237"} Nov 03 17:07:22.253832 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:07:22.253794 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:07:22.972689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:22.972648 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:07:22.988173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:22.988136 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:23.077376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.077292 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\") pod \"2b217f56-85d4-4515-9977-b58bebae9ae4\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " Nov 03 17:07:23.077376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.077358 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\") pod \"2b217f56-85d4-4515-9977-b58bebae9ae4\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " Nov 03 17:07:23.077663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.077425 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\" (UniqueName: \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\") pod \"2b217f56-85d4-4515-9977-b58bebae9ae4\" (UID: \"2b217f56-85d4-4515-9977-b58bebae9ae4\") " Nov 03 17:07:23.077834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.077808 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca" pod "2b217f56-85d4-4515-9977-b58bebae9ae4" (UID: "2b217f56-85d4-4515-9977-b58bebae9ae4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:23.078274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.078253 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.080023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.079995 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9" pod "2b217f56-85d4-4515-9977-b58bebae9ae4" (UID: "2b217f56-85d4-4515-9977-b58bebae9ae4"). InnerVolumeSpecName "kube-api-access-hdkf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:23.082755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.082723 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca" pod "2b217f56-85d4-4515-9977-b58bebae9ae4" (UID: "2b217f56-85d4-4515-9977-b58bebae9ae4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:07:23.180375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.180338 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b217f56-85d4-4515-9977-b58bebae9ae4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.180375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.180370 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdkf9\" (UniqueName: \"kubernetes.io/projected/2b217f56-85d4-4515-9977-b58bebae9ae4-kube-api-access-hdkf9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.522138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.522105 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" Nov 03 17:07:23.522138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.522131 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5" event={"ID":"2b217f56-85d4-4515-9977-b58bebae9ae4","Type":"ContainerDied","Data":"cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845"} Nov 03 17:07:23.522411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:23.522204 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba0192100a78bf6dd33c08416c46268d93e59d2bd2589db494cb45f6c131845" Nov 03 17:07:24.454526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:24.454464 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:07:24.455062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:24.454740 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" containerID="cri-o://e13b056792daba1e57c0a1e63157810dadd9c6fed03797b1e8b2a9820962e2dd" gracePeriod=30 Nov 03 17:07:25.905541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:25.905463 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:07:27.974086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:27.974036 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" podUID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" containerName="payload-processing" containerID="cri-o://5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b" gracePeriod=25 Nov 03 17:07:28.344574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.344538 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:07:28.490071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490030 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\") pod \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " Nov 03 17:07:28.490071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490074 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\") pod \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " Nov 03 17:07:28.490344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490225 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\") pod \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " Nov 03 17:07:28.490416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490395 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume" pod "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" (UID: "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:28.490498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490403 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\") pod \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\" (UID: \"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc\") " Nov 03 17:07:28.490733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.490708 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume" pod "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" (UID: "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:28.491409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.491379 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:28.491409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.491411 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:28.493218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.493188 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9" pod "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" (UID: "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc"). InnerVolumeSpecName "kube-api-access-4x7w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:28.494218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.494194 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials" pod "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" (UID: "1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:28.571945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.571833 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" containerID="5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b" exitCode=0 Nov 03 17:07:28.571945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.571907 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" Nov 03 17:07:28.572193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.571971 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" event={"ID":"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc","Type":"ContainerDied","Data":"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b"} Nov 03 17:07:28.572193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.572074 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt" event={"ID":"1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc","Type":"ContainerDied","Data":"9442da749a1145b9bf34eb5ce1e67f87aa39aab7a4dc7328329067975ed8288c"} Nov 03 17:07:28.572193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.572112 6001 scope.go:122] "RemoveContainer" containerID="5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b" Nov 03 17:07:28.588883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.588861 6001 scope.go:122] "RemoveContainer" containerID="5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b" Nov 03 17:07:28.589274 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:07:28.589248 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b\": container with ID starting with 5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b not found: ID does not exist" containerID="5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b" Nov 03 17:07:28.589391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.589285 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b"} err="failed to get container status \"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b\": rpc error: code = NotFound desc = could not find container \"5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b\": container with ID starting with 5ecd3aa1b5b4d25165a00ad2975cc5b046d1b5a9620a013f955b79eaa8a6fc5b not found: ID does not exist" Nov 03 17:07:28.593478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.593432 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:28.593478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.593480 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7w9\" (UniqueName: \"kubernetes.io/projected/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc-kube-api-access-4x7w9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:28.654470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.644358 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:07:28.654470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.651796 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b65fcjt"] Nov 03 17:07:28.866558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:28.858849 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" path="/var/lib/kubelet/pods/1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc/volumes" Nov 03 17:07:29.592419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.592382 6001 generic.go:356] "Generic (PLEG): container finished" podID="b05c6d79-8277-491b-a483-d5521eabbe31" containerID="e13b056792daba1e57c0a1e63157810dadd9c6fed03797b1e8b2a9820962e2dd" exitCode=0 Nov 03 17:07:29.593248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.592478 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" event={"ID":"b05c6d79-8277-491b-a483-d5521eabbe31","Type":"ContainerDied","Data":"e13b056792daba1e57c0a1e63157810dadd9c6fed03797b1e8b2a9820962e2dd"} Nov 03 17:07:29.748438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.748404 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:07:29.847491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.847382 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.847907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.847879 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.848109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.848091 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.848505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.848430 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.848678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.848660 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.848885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.848867 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.849021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849005 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.849219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849202 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.849384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849369 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\") pod \"b05c6d79-8277-491b-a483-d5521eabbe31\" (UID: \"b05c6d79-8277-491b-a483-d5521eabbe31\") " Nov 03 17:07:29.849519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.847904 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:29.849600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849078 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.849600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849118 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.849600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849282 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.849600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.849478 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.850695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.850671 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.850826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.850813 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05c6d79-8277-491b-a483-d5521eabbe31-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.850935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.850923 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.851031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.851018 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.851130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.851118 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.853392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.852667 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "kube-api-access-6jhkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:29.859407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.859383 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:29.859594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.859489 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:07:29.860542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.860519 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy" pod "b05c6d79-8277-491b-a483-d5521eabbe31" (UID: "b05c6d79-8277-491b-a483-d5521eabbe31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.960480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.954862 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jhkb\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-kube-api-access-6jhkb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.960480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.954933 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05c6d79-8277-491b-a483-d5521eabbe31-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.960480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.954950 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05c6d79-8277-491b-a483-d5521eabbe31-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.960480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:29.954963 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05c6d79-8277-491b-a483-d5521eabbe31-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:30.829721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.829676 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" event={"ID":"b05c6d79-8277-491b-a483-d5521eabbe31","Type":"ContainerDied","Data":"9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b"} Nov 03 17:07:30.830362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.830329 6001 scope.go:122] "RemoveContainer" containerID="e13b056792daba1e57c0a1e63157810dadd9c6fed03797b1e8b2a9820962e2dd" Nov 03 17:07:30.830566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.830542 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k" Nov 03 17:07:30.905439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.905403 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:07:30.905887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.905826 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" podUID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" containerName="payload-processing" containerID="cri-o://e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f" gracePeriod=25 Nov 03 17:07:30.915387 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:07:30.915352 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05c6d79_8277_491b_a483_d5521eabbe31.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05c6d79_8277_491b_a483_d5521eabbe31.slice/crio-9498ad1a53d9b60127e03ce6456d925251f02bae83d7169c6499004f8349244b\": RecentStats: unable to find data in memory cache]" Nov 03 17:07:30.917957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:30.917932 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-rrg2k"] Nov 03 17:07:31.232916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.232886 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:07:31.533468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.533043 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\") pod \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " Nov 03 17:07:31.533468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.533141 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\") pod \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " Nov 03 17:07:31.533468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.533363 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\") pod \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " Nov 03 17:07:31.533468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.533405 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\") pod \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\" (UID: \"5c40fb4c-5ffb-4239-9a08-ece4522920e9\") " Nov 03 17:07:31.533861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.533635 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume" pod "5c40fb4c-5ffb-4239-9a08-ece4522920e9" (UID: "5c40fb4c-5ffb-4239-9a08-ece4522920e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:31.552524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.534658 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:31.552524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.534956 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume" pod "5c40fb4c-5ffb-4239-9a08-ece4522920e9" (UID: "5c40fb4c-5ffb-4239-9a08-ece4522920e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:31.555782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.555644 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t" pod "5c40fb4c-5ffb-4239-9a08-ece4522920e9" (UID: "5c40fb4c-5ffb-4239-9a08-ece4522920e9"). InnerVolumeSpecName "kube-api-access-rfj6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:31.555782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.555748 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials" pod "5c40fb4c-5ffb-4239-9a08-ece4522920e9" (UID: "5c40fb4c-5ffb-4239-9a08-ece4522920e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:31.673475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.670522 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:31.673475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.670558 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfj6t\" (UniqueName: \"kubernetes.io/projected/5c40fb4c-5ffb-4239-9a08-ece4522920e9-kube-api-access-rfj6t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:31.673475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.670573 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c40fb4c-5ffb-4239-9a08-ece4522920e9-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:07:31.839859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.839776 6001 generic.go:356] "Generic (PLEG): container finished" podID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" containerID="e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f" exitCode=0 Nov 03 17:07:31.839859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.839825 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" Nov 03 17:07:31.840338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.839860 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" event={"ID":"5c40fb4c-5ffb-4239-9a08-ece4522920e9","Type":"ContainerDied","Data":"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f"} Nov 03 17:07:31.840338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.839916 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp" event={"ID":"5c40fb4c-5ffb-4239-9a08-ece4522920e9","Type":"ContainerDied","Data":"6e88277e0a3c81897ef37115879eb6716230fdde17c4fe72b2ecb6cdcc17bfb5"} Nov 03 17:07:31.840338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.839947 6001 scope.go:122] "RemoveContainer" containerID="e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f" Nov 03 17:07:31.855299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.855274 6001 scope.go:122] "RemoveContainer" containerID="e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f" Nov 03 17:07:31.855754 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:07:31.855728 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f\": container with ID starting with e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f not found: ID does not exist" containerID="e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f" Nov 03 17:07:31.855831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.855768 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f"} err="failed to get container status \"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f\": rpc error: code = NotFound desc = could not find container \"e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f\": container with ID starting with e46fd7c347502d2864caab5f9e91038ef94884165c2e240edf7f2b29fac0878f not found: ID does not exist" Nov 03 17:07:31.911739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.911706 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:07:31.922002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:31.921971 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677k6mbp"] Nov 03 17:07:32.791394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:32.791363 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" path="/var/lib/kubelet/pods/5c40fb4c-5ffb-4239-9a08-ece4522920e9/volumes" Nov 03 17:07:32.791960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:07:32.791943 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" path="/var/lib/kubelet/pods/b05c6d79-8277-491b-a483-d5521eabbe31/volumes" Nov 03 17:07:52.253523 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:07:52.253395 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:08:22.253687 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:08:22.253641 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:08:37.939609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.939567 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t"] Nov 03 17:08:37.940784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940758 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" containerName="payload-processing" Nov 03 17:08:37.940784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940782 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" containerName="payload-processing" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940803 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2b217f56-85d4-4515-9977-b58bebae9ae4" containerName="revoke-keys" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940811 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b217f56-85d4-4515-9977-b58bebae9ae4" containerName="revoke-keys" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940831 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" containerName="payload-processing" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940838 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" containerName="payload-processing" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940865 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" Nov 03 17:08:37.940985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.940872 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" Nov 03 17:08:37.941293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.941078 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2f4b92-79b6-4b2e-9970-cdd06fa95ebc" containerName="payload-processing" Nov 03 17:08:37.941293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.941095 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b217f56-85d4-4515-9977-b58bebae9ae4" containerName="revoke-keys" Nov 03 17:08:37.941293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.941110 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c40fb4c-5ffb-4239-9a08-ece4522920e9" containerName="payload-processing" Nov 03 17:08:37.941293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.941122 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05c6d79-8277-491b-a483-d5521eabbe31" containerName="istio-proxy" Nov 03 17:08:37.944350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.944327 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:37.960683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.960650 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:08:37.965616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.965585 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t"] Nov 03 17:08:37.973317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.973280 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:08:37.973317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.973280 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:08:37.973633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:37.973366 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:08:38.019929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.019848 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.020217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.020104 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.020264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.020220 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqv6h\" (UniqueName: \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.122344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.122305 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqv6h\" (UniqueName: \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.122544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.122533 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.122658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.122641 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.123299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.123273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.125003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.124971 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.146707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.146670 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqv6h\" (UniqueName: \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\") pod \"maas-api-revoke-sub-ee5f33d8-htn6t\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.259833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.259793 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:38.568429 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:08:38.568398 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eab1e11_c447_418b_9423_ec5e532f4067.slice/crio-3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1 WatchSource:0}: Error finding container 3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1: Status 404 returned error can't find the container with id 3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1 Nov 03 17:08:38.569593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.569564 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t"] Nov 03 17:08:38.589366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:38.589330 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" event={"ID":"0eab1e11-c447-418b-9423-ec5e532f4067","Type":"ContainerStarted","Data":"3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1"} Nov 03 17:08:39.602897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:39.602857 6001 generic.go:356] "Generic (PLEG): container finished" podID="0eab1e11-c447-418b-9423-ec5e532f4067" containerID="61e1005d56637ef30eb897ed86722845ceeb4113d8e9d5ae215ff7a2964e8130" exitCode=0 Nov 03 17:08:39.603354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:39.602893 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" event={"ID":"0eab1e11-c447-418b-9423-ec5e532f4067","Type":"ContainerDied","Data":"61e1005d56637ef30eb897ed86722845ceeb4113d8e9d5ae215ff7a2964e8130"} Nov 03 17:08:40.792266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.792238 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:40.983981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.983934 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\") pod \"0eab1e11-c447-418b-9423-ec5e532f4067\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " Nov 03 17:08:40.984227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.984032 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\" (UniqueName: \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\") pod \"0eab1e11-c447-418b-9423-ec5e532f4067\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " Nov 03 17:08:40.984491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.984438 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\") pod \"0eab1e11-c447-418b-9423-ec5e532f4067\" (UID: \"0eab1e11-c447-418b-9423-ec5e532f4067\") " Nov 03 17:08:40.984994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.984958 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca" pod "0eab1e11-c447-418b-9423-ec5e532f4067" (UID: "0eab1e11-c447-418b-9423-ec5e532f4067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:40.985367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.985341 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:40.986707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.986682 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h" pod "0eab1e11-c447-418b-9423-ec5e532f4067" (UID: "0eab1e11-c447-418b-9423-ec5e532f4067"). InnerVolumeSpecName "kube-api-access-lqv6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:40.987226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:40.987200 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca" pod "0eab1e11-c447-418b-9423-ec5e532f4067" (UID: "0eab1e11-c447-418b-9423-ec5e532f4067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:08:41.087574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:41.087535 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eab1e11-c447-418b-9423-ec5e532f4067-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:41.087574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:41.087568 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqv6h\" (UniqueName: \"kubernetes.io/projected/0eab1e11-c447-418b-9423-ec5e532f4067-kube-api-access-lqv6h\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:41.622529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:41.622496 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" Nov 03 17:08:41.622529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:41.622513 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t" event={"ID":"0eab1e11-c447-418b-9423-ec5e532f4067","Type":"ContainerDied","Data":"3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1"} Nov 03 17:08:41.622796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:41.622615 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d88cf2f184dba566062047428841fcd045a52a4d85932b13a7b01e1b2c9cac1" Nov 03 17:08:48.509903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.509864 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7"] Nov 03 17:08:48.510806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.510788 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0eab1e11-c447-418b-9423-ec5e532f4067" containerName="revoke-keys" Nov 03 17:08:48.510862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.510807 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eab1e11-c447-418b-9423-ec5e532f4067" containerName="revoke-keys" Nov 03 17:08:48.510971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.510962 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eab1e11-c447-418b-9423-ec5e532f4067" containerName="revoke-keys" Nov 03 17:08:48.514779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.514763 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.536406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.536374 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7"] Nov 03 17:08:48.542640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.542594 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:08:48.542640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.542608 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:08:48.542849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.542686 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:08:48.543115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.543098 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:08:48.620826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.620784 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.621030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.620932 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.621030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.620983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc9gg\" (UniqueName: \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.724467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.722852 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.724467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.722958 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc9gg\" (UniqueName: \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.724467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.723365 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.724796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.724553 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.727992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.727967 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.748857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.748829 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc9gg\" (UniqueName: \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\") pod \"maas-api-revoke-sub-86a2b42e-2pzm7\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:48.829822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:48.829756 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:49.138304 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:08:49.138275 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2030228a_dd14_48d4_a041_38dc73aa9b10.slice/crio-5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6 WatchSource:0}: Error finding container 5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6: Status 404 returned error can't find the container with id 5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6 Nov 03 17:08:49.140354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:49.140327 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7"] Nov 03 17:08:49.736285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:49.736247 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" event={"ID":"2030228a-dd14-48d4-a041-38dc73aa9b10","Type":"ContainerStarted","Data":"23b29ea0fb45e23078e63d21cd58a1a4d58c19849a50e888ab2fcaf1005e31c4"} Nov 03 17:08:49.736786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:49.736305 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" event={"ID":"2030228a-dd14-48d4-a041-38dc73aa9b10","Type":"ContainerStarted","Data":"5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6"} Nov 03 17:08:49.784578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:49.784528 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" podStartSLOduration=1.784515275 podStartE2EDuration="1.784515275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:08:49.782782507 +0000 UTC m=+3225.115996076" watchObservedRunningTime="2026-11-03 17:08:49.784515275 +0000 UTC m=+3225.117728844" Nov 03 17:08:50.747213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:50.747177 6001 generic.go:356] "Generic (PLEG): container finished" podID="2030228a-dd14-48d4-a041-38dc73aa9b10" containerID="23b29ea0fb45e23078e63d21cd58a1a4d58c19849a50e888ab2fcaf1005e31c4" exitCode=0 Nov 03 17:08:50.747672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:50.747233 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" event={"ID":"2030228a-dd14-48d4-a041-38dc73aa9b10","Type":"ContainerDied","Data":"23b29ea0fb45e23078e63d21cd58a1a4d58c19849a50e888ab2fcaf1005e31c4"} Nov 03 17:08:51.957512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:51.957485 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:52.080013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.079914 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\") pod \"2030228a-dd14-48d4-a041-38dc73aa9b10\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " Nov 03 17:08:52.080191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.080137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\") pod \"2030228a-dd14-48d4-a041-38dc73aa9b10\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " Nov 03 17:08:52.080250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.080185 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\" (UniqueName: \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\") pod \"2030228a-dd14-48d4-a041-38dc73aa9b10\" (UID: \"2030228a-dd14-48d4-a041-38dc73aa9b10\") " Nov 03 17:08:52.080556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.080524 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca" pod "2030228a-dd14-48d4-a041-38dc73aa9b10" (UID: "2030228a-dd14-48d4-a041-38dc73aa9b10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:52.080987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.080967 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:52.082162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.082138 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca" pod "2030228a-dd14-48d4-a041-38dc73aa9b10" (UID: "2030228a-dd14-48d4-a041-38dc73aa9b10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:08:52.082392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.082376 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg" pod "2030228a-dd14-48d4-a041-38dc73aa9b10" (UID: "2030228a-dd14-48d4-a041-38dc73aa9b10"). InnerVolumeSpecName "kube-api-access-lc9gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:52.182973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.182932 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc9gg\" (UniqueName: \"kubernetes.io/projected/2030228a-dd14-48d4-a041-38dc73aa9b10-kube-api-access-lc9gg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:52.182973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.182969 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2030228a-dd14-48d4-a041-38dc73aa9b10-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:08:52.252556 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:08:52.252517 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:08:52.764510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.764478 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" Nov 03 17:08:52.764700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.764476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7" event={"ID":"2030228a-dd14-48d4-a041-38dc73aa9b10","Type":"ContainerDied","Data":"5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6"} Nov 03 17:08:52.764700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:08:52.764629 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cf1d5c84b246f47261e2a29ff420c72f6975e2c22bdb3157955fb099c78ced6" Nov 03 17:09:01.380370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.380322 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7"] Nov 03 17:09:01.381539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.381512 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2030228a-dd14-48d4-a041-38dc73aa9b10" containerName="revoke-keys" Nov 03 17:09:01.381539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.381537 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2030228a-dd14-48d4-a041-38dc73aa9b10" containerName="revoke-keys" Nov 03 17:09:01.381789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.381774 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2030228a-dd14-48d4-a041-38dc73aa9b10" containerName="revoke-keys" Nov 03 17:09:01.386053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.386028 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.403506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.403472 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7"] Nov 03 17:09:01.410984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.410950 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:09:01.411146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.411014 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:09:01.444149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.444117 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:09:01.444419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.444226 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:09:01.507630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.507589 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.507834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.507672 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.507834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.507806 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjb2\" (UniqueName: \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.623475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.618482 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjb2\" (UniqueName: \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.623475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.618851 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.623475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.618982 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.623475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.621008 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.665594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.654594 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjb2\" (UniqueName: \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.673827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.654953 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0e53b5e-scpm7\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.702557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.702517 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:01.720988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.720955 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8"] Nov 03 17:09:01.728023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.727986 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:01.870649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.870602 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8"] Nov 03 17:09:01.923727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.923626 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x88g\" (UniqueName: \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:01.923961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.923931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:01.924190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:01.924156 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.026289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.026250 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.026542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.026428 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.026625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.026563 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x88g\" (UniqueName: \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.027533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.027507 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.029952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.029763 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.033406 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:02.033371 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eb019b_4ab4_4bd0_9a86_48aaf137d755.slice/crio-6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f WatchSource:0}: Error finding container 6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f: Status 404 returned error can't find the container with id 6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f Nov 03 17:09:02.035732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.035524 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7"] Nov 03 17:09:02.052861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.052828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x88g\" (UniqueName: \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\") pod \"maas-api-revoke-sub-e6e6c0e8-6k6j8\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.060350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.060324 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57"] Nov 03 17:09:02.066914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.066893 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.077319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.077301 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:02.086955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.086925 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57"] Nov 03 17:09:02.230373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.230340 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm7qz\" (UniqueName: \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.230643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.230620 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.230717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.230669 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.332246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.332211 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.332519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.332267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.332519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.332459 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm7qz\" (UniqueName: \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.333082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.333058 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.334777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.334757 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.359837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.359800 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm7qz\" (UniqueName: \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\") pod \"maas-api-revoke-sub-b8775f97-fxz57\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.385735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.385701 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:02.401948 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:02.401913 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ef1ad7_d9a4_402c_bbd7_eba49a40e39e.slice/crio-61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5 WatchSource:0}: Error finding container 61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5: Status 404 returned error can't find the container with id 61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5 Nov 03 17:09:02.403018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.402991 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8"] Nov 03 17:09:02.711850 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:02.711819 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75245d93_f369_4d19_aadc_2c8e0522f721.slice/crio-1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc WatchSource:0}: Error finding container 1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc: Status 404 returned error can't find the container with id 1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc Nov 03 17:09:02.713195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.713168 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57"] Nov 03 17:09:02.911131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.911092 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" event={"ID":"20eb019b-4ab4-4bd0-9a86-48aaf137d755","Type":"ContainerStarted","Data":"27e077b0d6a695c79cf776a7c860d6c0a03917d288e057fe2e0cab5cb264e5ea"} Nov 03 17:09:02.911341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.911159 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" event={"ID":"20eb019b-4ab4-4bd0-9a86-48aaf137d755","Type":"ContainerStarted","Data":"6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f"} Nov 03 17:09:02.912932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.912898 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" event={"ID":"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e","Type":"ContainerStarted","Data":"b6efbd876ecb42b3eec8afcbefa4ca8d043b83771087996b5d97f74f5bd28154"} Nov 03 17:09:02.913092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.912955 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" event={"ID":"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e","Type":"ContainerStarted","Data":"61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5"} Nov 03 17:09:02.914373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.914350 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" event={"ID":"75245d93-f369-4d19-aadc-2c8e0522f721","Type":"ContainerStarted","Data":"1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc"} Nov 03 17:09:02.957366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.957263 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" podStartSLOduration=1.957249096 podStartE2EDuration="1.957249096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:02.954488723 +0000 UTC m=+3238.287702286" watchObservedRunningTime="2026-11-03 17:09:02.957249096 +0000 UTC m=+3238.290462701" Nov 03 17:09:02.998990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:02.998935 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" podStartSLOduration=1.9989199420000001 podStartE2EDuration="1.998919942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:02.996905922 +0000 UTC m=+3238.330119492" watchObservedRunningTime="2026-11-03 17:09:02.998919942 +0000 UTC m=+3238.332133512" Nov 03 17:09:03.924899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.924807 6001 generic.go:356] "Generic (PLEG): container finished" podID="20eb019b-4ab4-4bd0-9a86-48aaf137d755" containerID="27e077b0d6a695c79cf776a7c860d6c0a03917d288e057fe2e0cab5cb264e5ea" exitCode=0 Nov 03 17:09:03.925312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.924892 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" event={"ID":"20eb019b-4ab4-4bd0-9a86-48aaf137d755","Type":"ContainerDied","Data":"27e077b0d6a695c79cf776a7c860d6c0a03917d288e057fe2e0cab5cb264e5ea"} Nov 03 17:09:03.926717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.926692 6001 generic.go:356] "Generic (PLEG): container finished" podID="52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" containerID="b6efbd876ecb42b3eec8afcbefa4ca8d043b83771087996b5d97f74f5bd28154" exitCode=0 Nov 03 17:09:03.926890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.926780 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" event={"ID":"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e","Type":"ContainerDied","Data":"b6efbd876ecb42b3eec8afcbefa4ca8d043b83771087996b5d97f74f5bd28154"} Nov 03 17:09:03.928606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.928584 6001 generic.go:356] "Generic (PLEG): container finished" podID="75245d93-f369-4d19-aadc-2c8e0522f721" containerID="5b5964c0fae80ee3fe6cd8fbe86de53dd55b0f3bee600430d9ef681d21debf71" exitCode=0 Nov 03 17:09:03.928750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:03.928635 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" event={"ID":"75245d93-f369-4d19-aadc-2c8e0522f721","Type":"ContainerDied","Data":"5b5964c0fae80ee3fe6cd8fbe86de53dd55b0f3bee600430d9ef681d21debf71"} Nov 03 17:09:05.199764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.199647 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:05.254515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.253001 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:05.283209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.283184 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:05.330243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330143 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\") pod \"75245d93-f369-4d19-aadc-2c8e0522f721\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " Nov 03 17:09:05.330426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330247 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\") pod \"75245d93-f369-4d19-aadc-2c8e0522f721\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " Nov 03 17:09:05.330426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330414 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\") pod \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " Nov 03 17:09:05.330593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330498 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\" (UniqueName: \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\") pod \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " Nov 03 17:09:05.330593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330557 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\") pod \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\" (UID: \"20eb019b-4ab4-4bd0-9a86-48aaf137d755\") " Nov 03 17:09:05.330729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.330617 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\" (UniqueName: \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\") pod \"75245d93-f369-4d19-aadc-2c8e0522f721\" (UID: \"75245d93-f369-4d19-aadc-2c8e0522f721\") " Nov 03 17:09:05.331570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.331339 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca" pod "20eb019b-4ab4-4bd0-9a86-48aaf137d755" (UID: "20eb019b-4ab4-4bd0-9a86-48aaf137d755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:05.331570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.331350 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca" pod "75245d93-f369-4d19-aadc-2c8e0522f721" (UID: "75245d93-f369-4d19-aadc-2c8e0522f721"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:05.331570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.331531 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.334427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.334401 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca" pod "20eb019b-4ab4-4bd0-9a86-48aaf137d755" (UID: "20eb019b-4ab4-4bd0-9a86-48aaf137d755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:05.334925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.334895 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2" pod "20eb019b-4ab4-4bd0-9a86-48aaf137d755" (UID: "20eb019b-4ab4-4bd0-9a86-48aaf137d755"). InnerVolumeSpecName "kube-api-access-tjjb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:05.335733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.335706 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz" pod "75245d93-f369-4d19-aadc-2c8e0522f721" (UID: "75245d93-f369-4d19-aadc-2c8e0522f721"). InnerVolumeSpecName "kube-api-access-pm7qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:05.342171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.342144 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca" pod "75245d93-f369-4d19-aadc-2c8e0522f721" (UID: "75245d93-f369-4d19-aadc-2c8e0522f721"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:05.432532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.432493 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\") pod \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " Nov 03 17:09:05.432690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.432554 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\") pod \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " Nov 03 17:09:05.432690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.432621 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\" (UniqueName: \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\") pod \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\" (UID: \"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e\") " Nov 03 17:09:05.432855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.432829 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca" pod "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" (UID: "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:05.433411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433385 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20eb019b-4ab4-4bd0-9a86-48aaf137d755-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.433411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433410 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjb2\" (UniqueName: \"kubernetes.io/projected/20eb019b-4ab4-4bd0-9a86-48aaf137d755-kube-api-access-tjjb2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.433630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433420 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm7qz\" (UniqueName: \"kubernetes.io/projected/75245d93-f369-4d19-aadc-2c8e0522f721-kube-api-access-pm7qz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.433630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433430 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.433630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433438 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.433630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.433462 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75245d93-f369-4d19-aadc-2c8e0522f721-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.434597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.434572 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca" pod "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" (UID: "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:05.434787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.434770 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g" pod "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" (UID: "52ef1ad7-d9a4-402c-bbd7-eba49a40e39e"). InnerVolumeSpecName "kube-api-access-7x88g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:05.534645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.534610 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.534645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.534646 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x88g\" (UniqueName: \"kubernetes.io/projected/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e-kube-api-access-7x88g\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:05.946679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.946588 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" Nov 03 17:09:05.946679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.946614 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57" event={"ID":"75245d93-f369-4d19-aadc-2c8e0522f721","Type":"ContainerDied","Data":"1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc"} Nov 03 17:09:05.946679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.946669 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1accaa92b495c8897e72365ed2b3f74d17a5a627e85aadb38b392915879084dc" Nov 03 17:09:05.948406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.948385 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" Nov 03 17:09:05.948565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.948419 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7" event={"ID":"20eb019b-4ab4-4bd0-9a86-48aaf137d755","Type":"ContainerDied","Data":"6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f"} Nov 03 17:09:05.948565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.948480 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6520486091a87ab5d7a6218f8cbd4908b4d80b160901be1b3b31761febf0480f" Nov 03 17:09:05.950137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.950120 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" Nov 03 17:09:05.950256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.950134 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8" event={"ID":"52ef1ad7-d9a4-402c-bbd7-eba49a40e39e","Type":"ContainerDied","Data":"61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5"} Nov 03 17:09:05.950256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:05.950176 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61c8160b5b0a2604156e978b791561f8b9897f52dab09558332e78c31ebebbe5" Nov 03 17:09:13.960554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.960517 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw"] Nov 03 17:09:13.961382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961363 6001 cpu_manager.go:414] "containerMap: removing container" podUID="52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" containerName="revoke-keys" Nov 03 17:09:13.961427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961383 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" containerName="revoke-keys" Nov 03 17:09:13.961427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961402 6001 cpu_manager.go:414] "containerMap: removing container" podUID="75245d93-f369-4d19-aadc-2c8e0522f721" containerName="revoke-keys" Nov 03 17:09:13.961427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961407 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75245d93-f369-4d19-aadc-2c8e0522f721" containerName="revoke-keys" Nov 03 17:09:13.961427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961418 6001 cpu_manager.go:414] "containerMap: removing container" podUID="20eb019b-4ab4-4bd0-9a86-48aaf137d755" containerName="revoke-keys" Nov 03 17:09:13.961427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961424 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20eb019b-4ab4-4bd0-9a86-48aaf137d755" containerName="revoke-keys" Nov 03 17:09:13.961619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961585 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="75245d93-f369-4d19-aadc-2c8e0522f721" containerName="revoke-keys" Nov 03 17:09:13.961619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961597 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="20eb019b-4ab4-4bd0-9a86-48aaf137d755" containerName="revoke-keys" Nov 03 17:09:13.961619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.961614 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" containerName="revoke-keys" Nov 03 17:09:13.965776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.965760 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:13.984609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.984579 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw"] Nov 03 17:09:13.991277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.991241 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:09:13.991496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.991350 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:09:13.991675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.991655 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:09:13.991738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:13.991715 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:09:14.069836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.069801 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.069836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.069839 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgg6p\" (UniqueName: \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.070136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.070111 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.175664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.175618 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.176634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.176606 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.191630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.191591 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgg6p\" (UniqueName: \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.191821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.177692 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.195258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.195230 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.220589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.220316 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgg6p\" (UniqueName: \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\") pod \"maas-api-revoke-sub-45686da4-4wzlw\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.285959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.285922 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:14.725482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.705244 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw"] Nov 03 17:09:14.755484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.746336 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7"] Nov 03 17:09:14.784459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.771023 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7"] Nov 03 17:09:14.784459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.771194 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:14.932468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.921858 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscmf\" (UniqueName: \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:14.932468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.922313 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:14.932468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:14.922392 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.038502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.036696 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.038502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.036793 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.038502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.036874 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscmf\" (UniqueName: \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.123466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.103751 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.262985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.117127 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.263254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.187505 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscmf\" (UniqueName: \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\") pod \"maas-api-revoke-sub-60079229-phkb7\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.267458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.267408 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:15.407731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.407636 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" event={"ID":"423bcfdc-07ce-4bcd-882a-d7fc2b668041","Type":"ContainerStarted","Data":"8937dd5584613f9e2696fc91f857992e5b737fcfe8a6c5f983e3ac53ad45d5bc"} Nov 03 17:09:15.747459 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:15.747415 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c4c9c9_98ec_4a59_8bed_7772d187fb77.slice/crio-c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07 WatchSource:0}: Error finding container c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07: Status 404 returned error can't find the container with id c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07 Nov 03 17:09:15.748644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:15.748620 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7"] Nov 03 17:09:16.418946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:16.418914 6001 generic.go:356] "Generic (PLEG): container finished" podID="423bcfdc-07ce-4bcd-882a-d7fc2b668041" containerID="b41c76b0a65ce8d7bc0478e4f9448210a48fcf93b3b3397682e3f41300486f16" exitCode=0 Nov 03 17:09:16.419376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:16.418998 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" event={"ID":"423bcfdc-07ce-4bcd-882a-d7fc2b668041","Type":"ContainerDied","Data":"b41c76b0a65ce8d7bc0478e4f9448210a48fcf93b3b3397682e3f41300486f16"} Nov 03 17:09:16.421182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:16.421153 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" event={"ID":"92c4c9c9-98ec-4a59-8bed-7772d187fb77","Type":"ContainerStarted","Data":"489af7573ecb3bbaf887919fd633b8663ed75a3fde0d54914c2d29c524b2a520"} Nov 03 17:09:16.421327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:16.421202 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" event={"ID":"92c4c9c9-98ec-4a59-8bed-7772d187fb77","Type":"ContainerStarted","Data":"c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07"} Nov 03 17:09:16.509754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:16.509686 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" podStartSLOduration=2.509669789 podStartE2EDuration="2.509669789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:16.506002498 +0000 UTC m=+3251.839216068" watchObservedRunningTime="2026-11-03 17:09:16.509669789 +0000 UTC m=+3251.842883368" Nov 03 17:09:17.431654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:17.431618 6001 generic.go:356] "Generic (PLEG): container finished" podID="92c4c9c9-98ec-4a59-8bed-7772d187fb77" containerID="489af7573ecb3bbaf887919fd633b8663ed75a3fde0d54914c2d29c524b2a520" exitCode=0 Nov 03 17:09:17.432079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:17.431704 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" event={"ID":"92c4c9c9-98ec-4a59-8bed-7772d187fb77","Type":"ContainerDied","Data":"489af7573ecb3bbaf887919fd633b8663ed75a3fde0d54914c2d29c524b2a520"} Nov 03 17:09:18.093417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.093379 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:18.202010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.201975 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\") pod \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " Nov 03 17:09:18.202221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.202107 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\") pod \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " Nov 03 17:09:18.202302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.202231 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\" (UniqueName: \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\") pod \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\" (UID: \"423bcfdc-07ce-4bcd-882a-d7fc2b668041\") " Nov 03 17:09:18.202539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.202514 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca" pod "423bcfdc-07ce-4bcd-882a-d7fc2b668041" (UID: "423bcfdc-07ce-4bcd-882a-d7fc2b668041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.202979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.202959 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.204389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.204359 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca" pod "423bcfdc-07ce-4bcd-882a-d7fc2b668041" (UID: "423bcfdc-07ce-4bcd-882a-d7fc2b668041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:18.204389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.204378 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p" pod "423bcfdc-07ce-4bcd-882a-d7fc2b668041" (UID: "423bcfdc-07ce-4bcd-882a-d7fc2b668041"). InnerVolumeSpecName "kube-api-access-wgg6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:18.304808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.304709 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/423bcfdc-07ce-4bcd-882a-d7fc2b668041-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.304808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.304747 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgg6p\" (UniqueName: \"kubernetes.io/projected/423bcfdc-07ce-4bcd-882a-d7fc2b668041-kube-api-access-wgg6p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.442216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.442185 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" Nov 03 17:09:18.442749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.442217 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw" event={"ID":"423bcfdc-07ce-4bcd-882a-d7fc2b668041","Type":"ContainerDied","Data":"8937dd5584613f9e2696fc91f857992e5b737fcfe8a6c5f983e3ac53ad45d5bc"} Nov 03 17:09:18.442749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.442269 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8937dd5584613f9e2696fc91f857992e5b737fcfe8a6c5f983e3ac53ad45d5bc" Nov 03 17:09:18.618745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.618715 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:18.711596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.711558 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\" (UniqueName: \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\") pod \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " Nov 03 17:09:18.711775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.711689 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\") pod \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " Nov 03 17:09:18.711775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.711759 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\") pod \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\" (UID: \"92c4c9c9-98ec-4a59-8bed-7772d187fb77\") " Nov 03 17:09:18.712629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.712592 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca" pod "92c4c9c9-98ec-4a59-8bed-7772d187fb77" (UID: "92c4c9c9-98ec-4a59-8bed-7772d187fb77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.714355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.714317 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca" pod "92c4c9c9-98ec-4a59-8bed-7772d187fb77" (UID: "92c4c9c9-98ec-4a59-8bed-7772d187fb77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:18.714537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.714429 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf" pod "92c4c9c9-98ec-4a59-8bed-7772d187fb77" (UID: "92c4c9c9-98ec-4a59-8bed-7772d187fb77"). InnerVolumeSpecName "kube-api-access-bscmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:18.815275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.815233 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscmf\" (UniqueName: \"kubernetes.io/projected/92c4c9c9-98ec-4a59-8bed-7772d187fb77-kube-api-access-bscmf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.815275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.815274 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.815509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:18.815290 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92c4c9c9-98ec-4a59-8bed-7772d187fb77-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:19.499636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:19.499601 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" event={"ID":"92c4c9c9-98ec-4a59-8bed-7772d187fb77","Type":"ContainerDied","Data":"c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07"} Nov 03 17:09:19.500049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:19.499651 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2a4b39d56f6d35bd903f3707292791725a40d33d9f741220bfaa3e93ff29c07" Nov 03 17:09:19.500049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:19.499615 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7" Nov 03 17:09:22.254309 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:09:22.254271 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:09:25.946625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.946493 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4"] Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.952691 6001 cpu_manager.go:414] "containerMap: removing container" podUID="92c4c9c9-98ec-4a59-8bed-7772d187fb77" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.952723 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c4c9c9-98ec-4a59-8bed-7772d187fb77" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.952782 6001 cpu_manager.go:414] "containerMap: removing container" podUID="423bcfdc-07ce-4bcd-882a-d7fc2b668041" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.952792 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423bcfdc-07ce-4bcd-882a-d7fc2b668041" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.953035 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="423bcfdc-07ce-4bcd-882a-d7fc2b668041" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.953069 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c4c9c9-98ec-4a59-8bed-7772d187fb77" containerName="revoke-keys" Nov 03 17:09:25.985563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.960593 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.005876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.993780 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:09:26.005876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.993919 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:09:26.005876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.993968 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:09:26.005876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:25.994020 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:09:26.013473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.009533 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4"] Nov 03 17:09:26.076984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.075943 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.076984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.076004 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75g7s\" (UniqueName: \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.076984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.076177 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.177391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.177355 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.177391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.177398 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75g7s\" (UniqueName: \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.177732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.177566 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.178098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.178078 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.180184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.180156 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.202172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.202093 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75g7s\" (UniqueName: \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\") pod \"maas-api-revoke-sub-ba903057-zrtv4\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.338013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.337976 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:26.707546 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:26.707497 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528d1ac4_2aa8_4301_aa8c_93cc88ff2f35.slice/crio-a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c WatchSource:0}: Error finding container a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c: Status 404 returned error can't find the container with id a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c Nov 03 17:09:26.709411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:26.709383 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4"] Nov 03 17:09:27.766530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:27.765186 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" event={"ID":"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35","Type":"ContainerStarted","Data":"dae35493b4b8eddf3550a427c447862193c7619476689708bb27f61cd3064e59"} Nov 03 17:09:27.766530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:27.765257 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" event={"ID":"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35","Type":"ContainerStarted","Data":"a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c"} Nov 03 17:09:27.837660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:27.837600 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" podStartSLOduration=2.837577231 podStartE2EDuration="2.837577231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:27.818880209 +0000 UTC m=+3263.152093779" watchObservedRunningTime="2026-11-03 17:09:27.837577231 +0000 UTC m=+3263.170790801" Nov 03 17:09:28.774766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:28.774725 6001 generic.go:356] "Generic (PLEG): container finished" podID="528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" containerID="dae35493b4b8eddf3550a427c447862193c7619476689708bb27f61cd3064e59" exitCode=0 Nov 03 17:09:28.775242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:28.774801 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" event={"ID":"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35","Type":"ContainerDied","Data":"dae35493b4b8eddf3550a427c447862193c7619476689708bb27f61cd3064e59"} Nov 03 17:09:29.999465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:29.999425 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:30.141695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.141535 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\") pod \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " Nov 03 17:09:30.142086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.142063 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\") pod \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " Nov 03 17:09:30.142423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.142404 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\" (UniqueName: \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\") pod \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\" (UID: \"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35\") " Nov 03 17:09:30.142974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.142461 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca" pod "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" (UID: "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:30.150717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.150688 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:30.164543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.164498 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca" pod "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" (UID: "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:30.172084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.172047 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s" pod "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" (UID: "528d1ac4-2aa8-4301-aa8c-93cc88ff2f35"). InnerVolumeSpecName "kube-api-access-75g7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:30.252695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.252653 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:30.252695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.252692 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75g7s\" (UniqueName: \"kubernetes.io/projected/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35-kube-api-access-75g7s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:30.809228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.809192 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" Nov 03 17:09:30.809475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.809428 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4" event={"ID":"528d1ac4-2aa8-4301-aa8c-93cc88ff2f35","Type":"ContainerDied","Data":"a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c"} Nov 03 17:09:30.809578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:30.809508 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a806f8ce9340c555a057b2578c6131552ba9c2ee929be50beae9c6847b0cc65c" Nov 03 17:09:37.739817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.739783 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7"] Nov 03 17:09:37.740672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.740656 6001 cpu_manager.go:414] "containerMap: removing container" podUID="528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" containerName="revoke-keys" Nov 03 17:09:37.740729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.740673 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" containerName="revoke-keys" Nov 03 17:09:37.740874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.740863 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" containerName="revoke-keys" Nov 03 17:09:37.744899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.744878 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.766901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.766865 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7"] Nov 03 17:09:37.768325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.768299 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:09:37.774547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.774518 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:09:37.774730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.774600 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:09:37.775487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.775462 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:09:37.865467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.863548 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qcw\" (UniqueName: \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.865467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.863678 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.865467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.863949 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.977469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.965694 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qcw\" (UniqueName: \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.979358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.979330 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.980286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.980265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:37.981399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:37.981126 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:38.040964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:38.040820 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:38.078469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:38.077633 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qcw\" (UniqueName: \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\") pod \"maas-api-revoke-sub-e65c5ac3-h2gl7\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:38.359794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:38.359705 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:38.668286 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:09:38.668258 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc48d37_2ec7_4036_ac13_e742278ae3b8.slice/crio-ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8 WatchSource:0}: Error finding container ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8: Status 404 returned error can't find the container with id ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8 Nov 03 17:09:38.670626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:38.670598 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7"] Nov 03 17:09:39.050007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:39.049970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" event={"ID":"ffc48d37-2ec7-4036-ac13-e742278ae3b8","Type":"ContainerStarted","Data":"ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8"} Nov 03 17:09:40.060040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:40.060003 6001 generic.go:356] "Generic (PLEG): container finished" podID="ffc48d37-2ec7-4036-ac13-e742278ae3b8" containerID="9144ce277d50b7b66c9b8a0552bebe351f02e0d3a37e09d93d7da963adf14fd6" exitCode=0 Nov 03 17:09:40.060523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:40.060074 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" event={"ID":"ffc48d37-2ec7-4036-ac13-e742278ae3b8","Type":"ContainerDied","Data":"9144ce277d50b7b66c9b8a0552bebe351f02e0d3a37e09d93d7da963adf14fd6"} Nov 03 17:09:41.283068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.283037 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:41.467586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.467481 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\") pod \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " Nov 03 17:09:41.467778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.467629 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\" (UniqueName: \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\") pod \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " Nov 03 17:09:41.467778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.467723 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\") pod \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\" (UID: \"ffc48d37-2ec7-4036-ac13-e742278ae3b8\") " Nov 03 17:09:41.467914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.467782 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca" pod "ffc48d37-2ec7-4036-ac13-e742278ae3b8" (UID: "ffc48d37-2ec7-4036-ac13-e742278ae3b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:41.468402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.468380 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:41.470423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.470398 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca" pod "ffc48d37-2ec7-4036-ac13-e742278ae3b8" (UID: "ffc48d37-2ec7-4036-ac13-e742278ae3b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:09:41.470598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.470438 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw" pod "ffc48d37-2ec7-4036-ac13-e742278ae3b8" (UID: "ffc48d37-2ec7-4036-ac13-e742278ae3b8"). InnerVolumeSpecName "kube-api-access-t4qcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:41.569689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.569645 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qcw\" (UniqueName: \"kubernetes.io/projected/ffc48d37-2ec7-4036-ac13-e742278ae3b8-kube-api-access-t4qcw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:41.569689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:41.569680 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc48d37-2ec7-4036-ac13-e742278ae3b8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:09:42.084359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:42.084328 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" Nov 03 17:09:42.084606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:42.084362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7" event={"ID":"ffc48d37-2ec7-4036-ac13-e742278ae3b8","Type":"ContainerDied","Data":"ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8"} Nov 03 17:09:42.084606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:09:42.084420 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed1432b6fb061d3f26be92005796d83d802e3e067904d28a135c575d2497d9e8" Nov 03 17:09:42.167689 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:09:42.167651 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc48d37_2ec7_4036_ac13_e742278ae3b8.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:09:52.253384 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:09:52.253338 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:10:01.058224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.058186 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2"] Nov 03 17:10:01.059380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.059353 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ffc48d37-2ec7-4036-ac13-e742278ae3b8" containerName="revoke-keys" Nov 03 17:10:01.059380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.059378 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc48d37-2ec7-4036-ac13-e742278ae3b8" containerName="revoke-keys" Nov 03 17:10:01.059692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.059675 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc48d37-2ec7-4036-ac13-e742278ae3b8" containerName="revoke-keys" Nov 03 17:10:01.064849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.064829 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.084134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.084103 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2"] Nov 03 17:10:01.089921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.089891 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:10:01.090129 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.089900 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:10:01.090315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.090017 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:10:01.096908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.096886 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:10:01.194882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.194841 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwbs\" (UniqueName: \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.195070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.194907 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.195267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.195246 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.297187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.297151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.297382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.297298 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwbs\" (UniqueName: \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.297382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.297356 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.298009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.297990 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.299723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.299703 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.321816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.321735 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwbs\" (UniqueName: \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\") pod \"maas-api-revoke-sub-5d2eebae-6zjz2\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.383522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.383486 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:01.702455 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:01.702405 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e428e1_d8d9_495e_a8d5_4c56d3c5aa60.slice/crio-dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2 WatchSource:0}: Error finding container dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2: Status 404 returned error can't find the container with id dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2 Nov 03 17:10:01.703947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.703917 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2"] Nov 03 17:10:01.971592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.971554 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68"] Nov 03 17:10:01.978696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:01.978670 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.008256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.008217 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68"] Nov 03 17:10:02.112552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.112517 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6t5\" (UniqueName: \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.112946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.112717 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.112946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.112893 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.226843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.226162 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.226843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.226367 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.226843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.226536 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6t5\" (UniqueName: \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.227439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.227408 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.229722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.229391 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.254199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.254163 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6t5\" (UniqueName: \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\") pod \"maas-api-revoke-sub-099f2358-drb68\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.297277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.297245 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:02.432974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.432923 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" event={"ID":"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60","Type":"ContainerStarted","Data":"3a3e48d15bbf57a77700ae5a65f9d7ace99276b33d8c8893f3b645069827707f"} Nov 03 17:10:02.433201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.433007 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" event={"ID":"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60","Type":"ContainerStarted","Data":"dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2"} Nov 03 17:10:02.621219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.617542 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" podStartSLOduration=1.617520866 podStartE2EDuration="1.617520866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:02.590852973 +0000 UTC m=+3297.924066542" watchObservedRunningTime="2026-11-03 17:10:02.617520866 +0000 UTC m=+3297.950734436" Nov 03 17:10:02.721059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:02.714423 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68"] Nov 03 17:10:02.732980 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:02.732940 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabc1056_32ae_47ed_9283_d70d59db4df5.slice/crio-dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715 WatchSource:0}: Error finding container dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715: Status 404 returned error can't find the container with id dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715 Nov 03 17:10:03.001982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.001947 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh"] Nov 03 17:10:03.007154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.007130 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.025163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.025132 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh"] Nov 03 17:10:03.114003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.112591 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxp2\" (UniqueName: \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.114003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.112801 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.114003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.112942 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.214186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.214085 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxp2\" (UniqueName: \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.214361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.214238 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.214361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.214338 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.215074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.215047 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.216751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.216729 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.239111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.239075 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxp2\" (UniqueName: \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\") pod \"maas-api-revoke-sub-af367c84-fpmwh\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.326674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.326632 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:03.447538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.447504 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" event={"ID":"babc1056-32ae-47ed-9283-d70d59db4df5","Type":"ContainerStarted","Data":"2be167a3a413ff7324155019ff83131d8ec2ca4facdd6d18a7483fa854d4bde9"} Nov 03 17:10:03.447738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.447568 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" event={"ID":"babc1056-32ae-47ed-9283-d70d59db4df5","Type":"ContainerStarted","Data":"dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715"} Nov 03 17:10:03.449968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.449932 6001 generic.go:356] "Generic (PLEG): container finished" podID="d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" containerID="3a3e48d15bbf57a77700ae5a65f9d7ace99276b33d8c8893f3b645069827707f" exitCode=0 Nov 03 17:10:03.450126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.450029 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" event={"ID":"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60","Type":"ContainerDied","Data":"3a3e48d15bbf57a77700ae5a65f9d7ace99276b33d8c8893f3b645069827707f"} Nov 03 17:10:03.502109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.502054 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" podStartSLOduration=2.502039023 podStartE2EDuration="2.502039023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:03.4986827 +0000 UTC m=+3298.831896270" watchObservedRunningTime="2026-11-03 17:10:03.502039023 +0000 UTC m=+3298.835252599" Nov 03 17:10:03.633817 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:03.633786 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e426c3_2da4_46b9_8f17_41103b42abbd.slice/crio-341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a WatchSource:0}: Error finding container 341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a: Status 404 returned error can't find the container with id 341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a Nov 03 17:10:03.636040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:03.636011 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh"] Nov 03 17:10:04.463341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.463286 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" event={"ID":"99e426c3-2da4-46b9-8f17-41103b42abbd","Type":"ContainerStarted","Data":"2ce75fefcef8d13ead4385987906165dccbebb07ebbd08ecf2017fc79c9b2a1e"} Nov 03 17:10:04.463822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.463398 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" event={"ID":"99e426c3-2da4-46b9-8f17-41103b42abbd","Type":"ContainerStarted","Data":"341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a"} Nov 03 17:10:04.465281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.465249 6001 generic.go:356] "Generic (PLEG): container finished" podID="babc1056-32ae-47ed-9283-d70d59db4df5" containerID="2be167a3a413ff7324155019ff83131d8ec2ca4facdd6d18a7483fa854d4bde9" exitCode=0 Nov 03 17:10:04.465413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.465334 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" event={"ID":"babc1056-32ae-47ed-9283-d70d59db4df5","Type":"ContainerDied","Data":"2be167a3a413ff7324155019ff83131d8ec2ca4facdd6d18a7483fa854d4bde9"} Nov 03 17:10:04.508944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.508879 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" podStartSLOduration=2.508859461 podStartE2EDuration="2.508859461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:04.506405194 +0000 UTC m=+3299.839618764" watchObservedRunningTime="2026-11-03 17:10:04.508859461 +0000 UTC m=+3299.842073031" Nov 03 17:10:04.743403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.743368 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:04.867054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.866954 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\") pod \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " Nov 03 17:10:04.867054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.867036 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\" (UniqueName: \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\") pod \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " Nov 03 17:10:04.867305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.867221 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\") pod \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\" (UID: \"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60\") " Nov 03 17:10:04.867305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.867276 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca" pod "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" (UID: "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:04.868073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.868051 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:04.869300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.869275 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca" pod "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" (UID: "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:10:04.869466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.869389 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs" pod "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" (UID: "d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60"). InnerVolumeSpecName "kube-api-access-5zwbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:04.996467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.986997 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zwbs\" (UniqueName: \"kubernetes.io/projected/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-kube-api-access-5zwbs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:04.996467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:04.987045 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:05.461169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.461142 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:10:05.475629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.475599 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" Nov 03 17:10:05.476086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.475636 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2" event={"ID":"d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60","Type":"ContainerDied","Data":"dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2"} Nov 03 17:10:05.476086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.475693 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3a2c19cc84cf7829d6d2e96cd5b152188b9af34608b1e04c3c333c659c7db2" Nov 03 17:10:05.477540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.477507 6001 generic.go:356] "Generic (PLEG): container finished" podID="99e426c3-2da4-46b9-8f17-41103b42abbd" containerID="2ce75fefcef8d13ead4385987906165dccbebb07ebbd08ecf2017fc79c9b2a1e" exitCode=0 Nov 03 17:10:05.477672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.477573 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" event={"ID":"99e426c3-2da4-46b9-8f17-41103b42abbd","Type":"ContainerDied","Data":"2ce75fefcef8d13ead4385987906165dccbebb07ebbd08ecf2017fc79c9b2a1e"} Nov 03 17:10:05.578735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.578699 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:10:05.676318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.676284 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:05.723403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.723371 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\" (UniqueName: \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\") pod \"babc1056-32ae-47ed-9283-d70d59db4df5\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " Nov 03 17:10:05.723733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.723708 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\") pod \"babc1056-32ae-47ed-9283-d70d59db4df5\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " Nov 03 17:10:05.723995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.723976 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\") pod \"babc1056-32ae-47ed-9283-d70d59db4df5\" (UID: \"babc1056-32ae-47ed-9283-d70d59db4df5\") " Nov 03 17:10:05.724571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.724429 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca" pod "babc1056-32ae-47ed-9283-d70d59db4df5" (UID: "babc1056-32ae-47ed-9283-d70d59db4df5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:05.725565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.725543 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:05.726601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.726574 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca" pod "babc1056-32ae-47ed-9283-d70d59db4df5" (UID: "babc1056-32ae-47ed-9283-d70d59db4df5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:10:05.726956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.726932 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5" pod "babc1056-32ae-47ed-9283-d70d59db4df5" (UID: "babc1056-32ae-47ed-9283-d70d59db4df5"). InnerVolumeSpecName "kube-api-access-nh6t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:05.827720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.827683 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh6t5\" (UniqueName: \"kubernetes.io/projected/babc1056-32ae-47ed-9283-d70d59db4df5-kube-api-access-nh6t5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:05.827720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:05.827717 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/babc1056-32ae-47ed-9283-d70d59db4df5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:06.573147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:06.573114 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" Nov 03 17:10:06.573663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:06.573151 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68" event={"ID":"babc1056-32ae-47ed-9283-d70d59db4df5","Type":"ContainerDied","Data":"dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715"} Nov 03 17:10:06.573663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:06.573239 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc26655a47706b116dc792b99e20dbd0e0ac5a439603d155eb411a7384eaa715" Nov 03 17:10:07.049976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.049949 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:07.096007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.095939 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\") pod \"99e426c3-2da4-46b9-8f17-41103b42abbd\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " Nov 03 17:10:07.096224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.096182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\") pod \"99e426c3-2da4-46b9-8f17-41103b42abbd\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " Nov 03 17:10:07.096316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.096276 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\" (UniqueName: \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\") pod \"99e426c3-2da4-46b9-8f17-41103b42abbd\" (UID: \"99e426c3-2da4-46b9-8f17-41103b42abbd\") " Nov 03 17:10:07.096384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.096342 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca" pod "99e426c3-2da4-46b9-8f17-41103b42abbd" (UID: "99e426c3-2da4-46b9-8f17-41103b42abbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:07.097113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.097089 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:07.099131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.099095 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca" pod "99e426c3-2da4-46b9-8f17-41103b42abbd" (UID: "99e426c3-2da4-46b9-8f17-41103b42abbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:10:07.099594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.099574 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2" pod "99e426c3-2da4-46b9-8f17-41103b42abbd" (UID: "99e426c3-2da4-46b9-8f17-41103b42abbd"). InnerVolumeSpecName "kube-api-access-wnxp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:07.198988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.198944 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e426c3-2da4-46b9-8f17-41103b42abbd-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:07.198988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.198986 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnxp2\" (UniqueName: \"kubernetes.io/projected/99e426c3-2da4-46b9-8f17-41103b42abbd-kube-api-access-wnxp2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:07.583178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.583146 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" Nov 03 17:10:07.583636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.583187 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh" event={"ID":"99e426c3-2da4-46b9-8f17-41103b42abbd","Type":"ContainerDied","Data":"341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a"} Nov 03 17:10:07.583636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:07.583245 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="341728a675c53b857f54299efaf3977b2da2fe78d8645d23c17ecceefe7d8a6a" Nov 03 17:10:11.961398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.961361 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj"] Nov 03 17:10:11.962338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962317 6001 cpu_manager.go:414] "containerMap: removing container" podUID="babc1056-32ae-47ed-9283-d70d59db4df5" containerName="revoke-keys" Nov 03 17:10:11.962400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962341 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babc1056-32ae-47ed-9283-d70d59db4df5" containerName="revoke-keys" Nov 03 17:10:11.962400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962359 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" containerName="revoke-keys" Nov 03 17:10:11.962400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962365 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" containerName="revoke-keys" Nov 03 17:10:11.962400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962374 6001 cpu_manager.go:414] "containerMap: removing container" podUID="99e426c3-2da4-46b9-8f17-41103b42abbd" containerName="revoke-keys" Nov 03 17:10:11.962400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962380 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e426c3-2da4-46b9-8f17-41103b42abbd" containerName="revoke-keys" Nov 03 17:10:11.962587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962528 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="babc1056-32ae-47ed-9283-d70d59db4df5" containerName="revoke-keys" Nov 03 17:10:11.962587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962539 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" containerName="revoke-keys" Nov 03 17:10:11.962587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.962550 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e426c3-2da4-46b9-8f17-41103b42abbd" containerName="revoke-keys" Nov 03 17:10:11.966637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.966618 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:11.990618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.990587 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj"] Nov 03 17:10:11.997956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.997928 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:10:11.997956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.997928 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:10:11.998216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.998008 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:10:11.998216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:11.998016 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:10:12.087121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.087084 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfwn\" (UniqueName: \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.087276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.087149 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.087494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.087472 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.188766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.188721 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfwn\" (UniqueName: \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.188947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.188865 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.189138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.189122 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.189670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.189646 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.191580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.191557 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.213241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.213155 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfwn\" (UniqueName: \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\") pod \"maas-api-revoke-sub-bd0fe45b-zpwmj\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.282703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.282668 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:12.593499 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:12.593466 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41669dd_d2d5_414f_be11_a53541443ae6.slice/crio-b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746 WatchSource:0}: Error finding container b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746: Status 404 returned error can't find the container with id b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746 Nov 03 17:10:12.596121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.596091 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj"] Nov 03 17:10:12.629667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:12.629627 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" event={"ID":"a41669dd-d2d5-414f-be11-a53541443ae6","Type":"ContainerStarted","Data":"b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746"} Nov 03 17:10:13.688103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:13.688050 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" event={"ID":"a41669dd-d2d5-414f-be11-a53541443ae6","Type":"ContainerStarted","Data":"84ca818aa2cae79b15564b246d38e8e61280e09d2d514e7e3291edd987947c04"} Nov 03 17:10:13.733580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:13.733517 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" podStartSLOduration=2.733501344 podStartE2EDuration="2.733501344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:13.73188967 +0000 UTC m=+3309.065103245" watchObservedRunningTime="2026-11-03 17:10:13.733501344 +0000 UTC m=+3309.066714940" Nov 03 17:10:14.698404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:14.698365 6001 generic.go:356] "Generic (PLEG): container finished" podID="a41669dd-d2d5-414f-be11-a53541443ae6" containerID="84ca818aa2cae79b15564b246d38e8e61280e09d2d514e7e3291edd987947c04" exitCode=0 Nov 03 17:10:14.698984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:14.698473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" event={"ID":"a41669dd-d2d5-414f-be11-a53541443ae6","Type":"ContainerDied","Data":"84ca818aa2cae79b15564b246d38e8e61280e09d2d514e7e3291edd987947c04"} Nov 03 17:10:15.607086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.607019 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25"] Nov 03 17:10:15.646044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.645259 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.646344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.646325 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25"] Nov 03 17:10:15.818471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.802408 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzdc\" (UniqueName: \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.818471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.802614 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.818471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.802728 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.960530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.954929 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.960530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.955058 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:15.960530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.955336 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzdc\" (UniqueName: \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:16.004670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:15.996463 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:16.141284 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.141250 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:10:16.155472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.154111 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.165269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.165217 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:10:16.168692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.168638 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:16.175790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.175756 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzdc\" (UniqueName: \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\") pod \"maas-api-revoke-sub-6fd35899-sts25\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:16.231984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.231949 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.231998 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232023 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ww5p\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232063 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232113 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232182 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232324 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.232377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.232368 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.268082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.268038 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:16.334257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334269 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ww5p\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334325 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334414 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334811 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.334892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334912 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.334981 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.335103 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.335133 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.335248 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.335357 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.335588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.335524 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.337500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.337432 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.337843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.337820 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.352536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.352506 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:16.361075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.361011 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ww5p\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.361075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.361054 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-twb8c\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.436162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.436130 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\") pod \"a41669dd-d2d5-414f-be11-a53541443ae6\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " Nov 03 17:10:16.436370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.436308 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\") pod \"a41669dd-d2d5-414f-be11-a53541443ae6\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " Nov 03 17:10:16.436465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.436420 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\" (UniqueName: \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\") pod \"a41669dd-d2d5-414f-be11-a53541443ae6\" (UID: \"a41669dd-d2d5-414f-be11-a53541443ae6\") " Nov 03 17:10:16.436565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.436533 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca" pod "a41669dd-d2d5-414f-be11-a53541443ae6" (UID: "a41669dd-d2d5-414f-be11-a53541443ae6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:16.437340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.437316 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:16.438849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.438818 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca" pod "a41669dd-d2d5-414f-be11-a53541443ae6" (UID: "a41669dd-d2d5-414f-be11-a53541443ae6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:10:16.439255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.439227 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn" pod "a41669dd-d2d5-414f-be11-a53541443ae6" (UID: "a41669dd-d2d5-414f-be11-a53541443ae6"). InnerVolumeSpecName "kube-api-access-2mfwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:16.505537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.505429 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:16.539364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.539327 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a41669dd-d2d5-414f-be11-a53541443ae6-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:16.539364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.539361 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mfwn\" (UniqueName: \"kubernetes.io/projected/a41669dd-d2d5-414f-be11-a53541443ae6-kube-api-access-2mfwn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:16.592421 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:16.592373 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc8532f_a07c_4360_90f0_38e4acd87103.slice/crio-b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116 WatchSource:0}: Error finding container b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116: Status 404 returned error can't find the container with id b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116 Nov 03 17:10:16.595485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.595272 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25"] Nov 03 17:10:16.867454 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:16.867411 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ab24eb_f971_4ff2_93c9_8a6539ed40db.slice/crio-0deb7e8585eb1e187499b12cca964b0568796389fb36423c66e68ae58e1f1d32 WatchSource:0}: Error finding container 0deb7e8585eb1e187499b12cca964b0568796389fb36423c66e68ae58e1f1d32: Status 404 returned error can't find the container with id 0deb7e8585eb1e187499b12cca964b0568796389fb36423c66e68ae58e1f1d32 Nov 03 17:10:16.870319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.870288 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:10:16.871713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.871677 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:16.871816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.871750 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:16.871816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.871783 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:16.882043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.882007 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" event={"ID":"5bc8532f-a07c-4360-90f0-38e4acd87103","Type":"ContainerStarted","Data":"b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116"} Nov 03 17:10:16.883555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.883527 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" event={"ID":"d5ab24eb-f971-4ff2-93c9-8a6539ed40db","Type":"ContainerStarted","Data":"0deb7e8585eb1e187499b12cca964b0568796389fb36423c66e68ae58e1f1d32"} Nov 03 17:10:16.885064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.885045 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" Nov 03 17:10:16.885165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.885084 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj" event={"ID":"a41669dd-d2d5-414f-be11-a53541443ae6","Type":"ContainerDied","Data":"b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746"} Nov 03 17:10:16.885165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:16.885129 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0a8d15a29b34602e6fe139d1dcd272a26b7adf2945a1280ca0b54820a0ff746" Nov 03 17:10:17.900332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:17.900289 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" event={"ID":"d5ab24eb-f971-4ff2-93c9-8a6539ed40db","Type":"ContainerStarted","Data":"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b"} Nov 03 17:10:17.902389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:17.902361 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" event={"ID":"5bc8532f-a07c-4360-90f0-38e4acd87103","Type":"ContainerStarted","Data":"2b62e0a7117071872d4e2c4fb3c6936365780cf8023e5e544542422be759b7c8"} Nov 03 17:10:17.974728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:17.974661 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podStartSLOduration=2.974630517 podStartE2EDuration="2.974630517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:17.972855073 +0000 UTC m=+3313.306068669" watchObservedRunningTime="2026-11-03 17:10:17.974630517 +0000 UTC m=+3313.307844087" Nov 03 17:10:18.021738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.021685 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" podStartSLOduration=3.021669244 podStartE2EDuration="3.021669244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:18.016788542 +0000 UTC m=+3313.350002111" watchObservedRunningTime="2026-11-03 17:10:18.021669244 +0000 UTC m=+3313.354882813" Nov 03 17:10:18.506140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.506100 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:18.507700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.507669 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:18.507876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.507755 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:18.912566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.912468 6001 generic.go:356] "Generic (PLEG): container finished" podID="5bc8532f-a07c-4360-90f0-38e4acd87103" containerID="2b62e0a7117071872d4e2c4fb3c6936365780cf8023e5e544542422be759b7c8" exitCode=0 Nov 03 17:10:18.912566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:18.912539 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" event={"ID":"5bc8532f-a07c-4360-90f0-38e4acd87103","Type":"ContainerDied","Data":"2b62e0a7117071872d4e2c4fb3c6936365780cf8023e5e544542422be759b7c8"} Nov 03 17:10:19.505864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:19.505830 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:19.506054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:19.505882 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:20.130872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.130845 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.226841 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\") pod \"5bc8532f-a07c-4360-90f0-38e4acd87103\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.227117 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\" (UniqueName: \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\") pod \"5bc8532f-a07c-4360-90f0-38e4acd87103\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.227225 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\") pod \"5bc8532f-a07c-4360-90f0-38e4acd87103\" (UID: \"5bc8532f-a07c-4360-90f0-38e4acd87103\") " Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.227718 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca" pod "5bc8532f-a07c-4360-90f0-38e4acd87103" (UID: "5bc8532f-a07c-4360-90f0-38e4acd87103"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.228383 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.230227 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc" pod "5bc8532f-a07c-4360-90f0-38e4acd87103" (UID: "5bc8532f-a07c-4360-90f0-38e4acd87103"). InnerVolumeSpecName "kube-api-access-5rzdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:20.235637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.230366 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca" pod "5bc8532f-a07c-4360-90f0-38e4acd87103" (UID: "5bc8532f-a07c-4360-90f0-38e4acd87103"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:10:20.329482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.329435 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzdc\" (UniqueName: \"kubernetes.io/projected/5bc8532f-a07c-4360-90f0-38e4acd87103-kube-api-access-5rzdc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:20.329482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.329477 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bc8532f-a07c-4360-90f0-38e4acd87103-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:10:20.505921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.505829 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:20.505921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.505884 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:20.931748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.931707 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" event={"ID":"5bc8532f-a07c-4360-90f0-38e4acd87103","Type":"ContainerDied","Data":"b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116"} Nov 03 17:10:20.931748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.931725 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25" Nov 03 17:10:20.932004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:20.931761 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b05606f2a0e8d5560fc129293442b4d3b155196a6b43f26484426044eaa3e116" Nov 03 17:10:21.506285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:21.506246 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:21.506765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:21.506309 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:22.300158 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:10:22.300112 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:10:22.505920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:22.505885 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:22.506111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:22.505943 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:23.506404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:23.506361 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:23.507013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:23.506430 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:24.506994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:24.506950 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:24.507436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:24.507028 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:25.138472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:25.131906 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj"] Nov 03 17:10:25.179903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:25.179822 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6232b9b7-lggrj"] Nov 03 17:10:25.506229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:25.506189 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 17:10:25.506495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:25.506261 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 17:10:26.806041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:26.805999 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c523f1-ae7e-4ef4-970b-8b11f2575511" path="/var/lib/kubelet/pods/c2c523f1-ae7e-4ef4-970b-8b11f2575511/volumes" Nov 03 17:10:27.509049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:27.509001 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:27.509288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:27.509065 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:28.550682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:28.550634 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:28.551164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:28.550720 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:29.711464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:29.711411 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-twb8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:29.712012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:29.711513 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:30.178116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.178078 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:30.178499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.178472 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:30.179259 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.179240 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:10:30.947476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.946116 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:10:30.947476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947458 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a41669dd-d2d5-414f-be11-a53541443ae6" containerName="revoke-keys" Nov 03 17:10:30.947476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947481 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41669dd-d2d5-414f-be11-a53541443ae6" containerName="revoke-keys" Nov 03 17:10:30.948164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947540 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5bc8532f-a07c-4360-90f0-38e4acd87103" containerName="revoke-keys" Nov 03 17:10:30.948164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947551 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bc8532f-a07c-4360-90f0-38e4acd87103" containerName="revoke-keys" Nov 03 17:10:30.948164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947799 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41669dd-d2d5-414f-be11-a53541443ae6" containerName="revoke-keys" Nov 03 17:10:30.948164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:30.947820 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bc8532f-a07c-4360-90f0-38e4acd87103" containerName="revoke-keys" Nov 03 17:10:31.030255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.030177 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.104401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.104359 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:10:31.111896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.111824 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.113241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.113208 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.113487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.113464 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.113688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.113668 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.113855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.113838 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.114008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.113991 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.114188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.114169 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.114607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.114585 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2xh\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.114736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.114714 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.116910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.115581 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.175586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.175550 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:10:31.179691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.179614 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:10:31.235168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.219198 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235495 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235552 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235614 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235701 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235736 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235784 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235938 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236013 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95pc\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236056 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236094 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2xh\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236153 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236282 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236394 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236436 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236503 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.261474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.236529 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.288056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.235077 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.289305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.289271 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.290512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.290484 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.310689 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.310917 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.325250 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.336997 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.338361 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401651 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401701 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401784 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401852 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95pc\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.401971 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.402038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.402258 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.367842 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.402857 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.413545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.368600 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.414902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.414866 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.428667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.415529 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.428667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.381341 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2xh\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-gd4ql\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.428667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.416098 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.428667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.417838 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.442132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.442097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.462565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.458948 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95pc\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.487253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.473389 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-944ln\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.487533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.477862 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:31.784883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.784780 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:31.884652 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:31.884605 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42248da_ec4d_4810_b331_250069ca99c8.slice/crio-f55c72b2eeb2bf04d521f66a625ad9840cef60e22fa96d862a1b393585d26297 WatchSource:0}: Error finding container f55c72b2eeb2bf04d521f66a625ad9840cef60e22fa96d862a1b393585d26297: Status 404 returned error can't find the container with id f55c72b2eeb2bf04d521f66a625ad9840cef60e22fa96d862a1b393585d26297 Nov 03 17:10:31.887541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.887497 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:31.887662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.887597 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:31.887662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.887644 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:31.891669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:31.891649 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:10:32.111118 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:10:32.111084 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9b74e6_ff47_457e_8806_781259a6d687.slice/crio-92c2cb66005f7ce3097e7f9025e03c9083846647ca185f7d7bfd3f5edcbeeda2 WatchSource:0}: Error finding container 92c2cb66005f7ce3097e7f9025e03c9083846647ca185f7d7bfd3f5edcbeeda2: Status 404 returned error can't find the container with id 92c2cb66005f7ce3097e7f9025e03c9083846647ca185f7d7bfd3f5edcbeeda2 Nov 03 17:10:32.112967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.112935 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:10:32.115128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.115087 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:32.115254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.115162 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:32.115254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.115195 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:10:32.623707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.621549 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" event={"ID":"a42248da-ec4d-4810-b331-250069ca99c8","Type":"ContainerStarted","Data":"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee"} Nov 03 17:10:32.623707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.621622 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" event={"ID":"a42248da-ec4d-4810-b331-250069ca99c8","Type":"ContainerStarted","Data":"f55c72b2eeb2bf04d521f66a625ad9840cef60e22fa96d862a1b393585d26297"} Nov 03 17:10:32.653476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.653413 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" event={"ID":"6d9b74e6-ff47-457e-8806-781259a6d687","Type":"ContainerStarted","Data":"92c2cb66005f7ce3097e7f9025e03c9083846647ca185f7d7bfd3f5edcbeeda2"} Nov 03 17:10:32.709659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:32.709553 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podStartSLOduration=2.709532308 podStartE2EDuration="2.709532308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:32.698047805 +0000 UTC m=+3328.031261375" watchObservedRunningTime="2026-11-03 17:10:32.709532308 +0000 UTC m=+3328.042745879" Nov 03 17:10:33.480730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.480676 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:33.484225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.484192 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:33.484544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.484500 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:33.694370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.692145 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" event={"ID":"6d9b74e6-ff47-457e-8806-781259a6d687","Type":"ContainerStarted","Data":"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b"} Nov 03 17:10:33.779970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.778232 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podStartSLOduration=3.778210065 podStartE2EDuration="3.778210065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:33.759159766 +0000 UTC m=+3329.092373349" watchObservedRunningTime="2026-11-03 17:10:33.778210065 +0000 UTC m=+3329.111423635" Nov 03 17:10:33.799472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.795124 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:33.799472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.796655 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:33.799472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:33.796710 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:34.478983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:34.478945 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:34.479346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:34.479307 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:34.813486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:34.812916 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:34.813486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:34.812989 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:35.479726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.479681 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:35.480025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.479988 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:35.517767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.517722 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded" Nov 03 17:10:35.804962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.804866 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:35.804962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.804943 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:35.942938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:35.942886 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.029598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.029549 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.057600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.057477 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-286sk" podUID="045a4032-eeeb-46f7-a7f5-ebbbdc01558a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.26:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.152693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.152646 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 17:10:36.218835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.218770 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.254904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.253660 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.322853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.322711 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 17:10:36.441013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.440562 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.486764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.486269 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:36.486764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.486333 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:36.848477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.842947 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:36.848477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.843009 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:36.947477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.943663 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.985479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:36.964315 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:10:37.031786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.030477 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:37.031786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.031430 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:10:37.065979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.055900 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:10:37.184533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.183753 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 17:10:37.231468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.200272 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:10:37.685708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.685670 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:37.686013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.685978 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:37.825469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.802782 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:37.825469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:37.802843 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:38.035384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.035338 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:38.080680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.080625 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:38.297777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.297638 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:38.478501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.458379 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:38.495538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.465673 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:38.544470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.465714 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 17:10:38.546930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.465755 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 17:10:38.609261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.499665 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:38.628623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.609551 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:38.782465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.768330 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:10:38.906534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.800615 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:38.944470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.923770 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:38.967061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:38.967000 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.264743 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.264525 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.301826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.300870 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.455476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.444502 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:10:39.481474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.481412 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.486493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.482811 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:10:39.514927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.489817 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:39.515496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.515458 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:39.529578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.509551 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:10:39.530228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.530198 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:10:39.558468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.548528 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.558468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.548969 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:10:39.623896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.623840 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.635260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.635226 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 17:10:39.908542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.871359 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:39.908542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.895016 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:39.908542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.895070 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:39.956472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:39.951955 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" Nov 03 17:10:40.296480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.283357 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 17:10:40.359078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.302818 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:40.375468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.371784 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:40.448505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.437915 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:10:40.470470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.465603 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:10:40.523466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.517893 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:10:40.523690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.523535 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:10:40.523690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.523649 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:40.523843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.523728 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:40.528561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.528528 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:10:40.798470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.791648 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:40.798470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:40.791728 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:41.478635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:41.478592 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:41.484466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:41.479300 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:41.785949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:41.785642 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:41.786222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:41.786190 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:42.478547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:42.478506 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:42.478759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:42.478579 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:42.787492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:42.787376 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:42.787699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:42.787497 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:43.485098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:43.485060 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" start-of-body= Nov 03 17:10:43.485764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:43.485730 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": dial tcp 10.134.1.50:15021: connect: connection refused" Nov 03 17:10:43.787643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:43.787543 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Nov 03 17:10:43.787968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:43.787927 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Nov 03 17:10:45.535960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:45.534876 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:45.535960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:45.534955 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:45.848898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:45.848799 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-944ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:10:45.849199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:45.849154 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:46.231512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.230668 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:46.233076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.232932 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:46.235063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.235026 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:10:46.464468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.463836 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:46.468466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.466524 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:46.473463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:46.472770 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:10:47.859469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:47.852304 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:47.963032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:10:47.962548 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:52.275187 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:10:52.275145 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:11:04.274677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.274631 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:11:04.280752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.280720 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.302019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.301984 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:11:04.308776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.308743 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-da409b-openshift-default-dockercfg-4js9c\"" Nov 03 17:11:04.373277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.373234 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.373687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.373636 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.374547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.374519 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.375076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.375049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.375310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.375294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.375486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.375466 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.375965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.375943 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.376487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.376462 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kqf6\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.376941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.376922 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.478884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.478844 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.478925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.478955 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.478971 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.478996 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479065 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kqf6\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479112 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479396 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479524 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479645 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479748 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.479854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479803 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.480004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.479986 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.480085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.480046 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.485258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.485233 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.485491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.485468 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.505980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.505949 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.506166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.506039 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kqf6\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\") pod \"e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:04.599709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:04.599615 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:05.672757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:05.671332 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:11:05.676857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:05.676809 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:11:05.677029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:05.676918 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:11:05.677029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:05.676967 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:11:06.800468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:06.791430 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" event={"ID":"1425396b-f582-4528-a8c9-0b2969cfb8c0","Type":"ContainerStarted","Data":"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559"} Nov 03 17:11:06.800468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:06.791514 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" event={"ID":"1425396b-f582-4528-a8c9-0b2969cfb8c0","Type":"ContainerStarted","Data":"75ab90e6127de380a319e57638c97fc70247e9d2991c1080e4a4d0c1f55623cc"} Nov 03 17:11:06.932465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:06.929462 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podStartSLOduration=2.929423577 podStartE2EDuration="2.929423577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:11:06.886659218 +0000 UTC m=+3362.219872782" watchObservedRunningTime="2026-11-03 17:11:06.929423577 +0000 UTC m=+3362.262637147" Nov 03 17:11:07.607470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:07.601536 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:07.607470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:07.601674 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:07.607470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:07.601724 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:08.617876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:08.617833 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:08.618569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:08.618527 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:09.303610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.303551 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:09.379517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.379457 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:09.525682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.525633 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:09.526463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.526408 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:09.604990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.604887 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:09.605267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.605234 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:09.627160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.627115 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 17:11:09.794471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.775223 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:09.965590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:09.965461 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:10.305603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.305548 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:10.306536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.306507 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:11:10.371603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.371549 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:10.530865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.530823 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 17:11:10.531684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.531658 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:11:10.537298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.537256 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:10.537971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.537559 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 17:11:10.538465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.538178 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:11:10.538672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.538656 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:11:10.600576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.600429 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:10.600906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.600848 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:10.629606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.629562 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:11:10.897962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.731840 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:11:10.899231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.899203 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:11:10.899496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.899481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:11:10.907474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:10.806486 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:11:11.201576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.201251 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:11:11.201576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.201402 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:11:11.201576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.201436 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:11:11.201576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.201482 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:11:11.201576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.201523 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:11:11.600729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.600682 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:11.600931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:11.600768 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:12.601029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:12.600988 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:12.601527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:12.601052 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:13.600742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:13.600699 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" start-of-body= Nov 03 17:11:13.600948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:13.600761 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.52:15021/healthz/ready\": dial tcp 10.134.1.52:15021: connect: connection refused" Nov 03 17:11:14.605380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:14.605335 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:14.605917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:14.605831 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:14.606436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:14.606418 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:11:22.254698 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:11:22.254656 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:11:42.032281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:42.032236 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6"] Nov 03 17:11:42.046999 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:42.046963 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96f09a19-h8nf6"] Nov 03 17:11:42.792537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:42.792492 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b6009a3-2284-47ba-8c9a-3fb78c53e2e0" path="/var/lib/kubelet/pods/0b6009a3-2284-47ba-8c9a-3fb78c53e2e0/volumes" Nov 03 17:11:52.252412 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:11:52.252373 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:11:53.048013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:53.047979 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6"] Nov 03 17:11:53.060142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:53.060105 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35db2b49-rqfl6"] Nov 03 17:11:54.794532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:11:54.794499 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103" path="/var/lib/kubelet/pods/d5f0b10d-ee03-4dfb-9fff-41ba0b4a6103/volumes" Nov 03 17:12:03.420175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.420142 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:12:03.425684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.425663 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.441769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.441727 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6046c2\"" Nov 03 17:12:03.447770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.447744 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:12:03.454755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.454733 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"routing-overlay\"" Nov 03 17:12:03.454876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.454733 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"openshift-service-ca.crt\"" Nov 03 17:12:03.454876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.454808 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"kube-root-ca.crt\"" Nov 03 17:12:03.455011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.454928 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"agc-cred-9b784297c053\"" Nov 03 17:12:03.455011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.454999 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-e2e--40ad01b7-dockercfg-6dhrc\"" Nov 03 17:12:03.503272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.503238 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.503468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.503360 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.503468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.503433 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm8cq\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.503599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.503561 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.606468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.605733 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.606468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.605925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.606468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.606029 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm8cq\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.606468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.606163 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.608292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.607545 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.638549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.620203 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.646466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.644770 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.656468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.648404 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm8cq\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:03.752020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:03.751980 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:04.171327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:04.171302 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:12:04.987422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:04.987378 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" event={"ID":"a5f9da16-e423-4203-a8f7-e275b8b435eb","Type":"ContainerStarted","Data":"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083"} Nov 03 17:12:04.987880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:04.987438 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" event={"ID":"a5f9da16-e423-4203-a8f7-e275b8b435eb","Type":"ContainerStarted","Data":"71f32e1e4d1f3ee7b7901572d0f17b6bd997f6ab8fdd6261971bcddabacc9e9b"} Nov 03 17:12:04.987880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:04.987529 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:05.033077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:05.033026 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" podStartSLOduration=2.033013443 podStartE2EDuration="2.033013443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:05.03065358 +0000 UTC m=+3420.363867151" watchObservedRunningTime="2026-11-03 17:12:05.033013443 +0000 UTC m=+3420.366227012" Nov 03 17:12:06.033131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.033099 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls"] Nov 03 17:12:06.048238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.048207 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d192a6c8-xw5ls"] Nov 03 17:12:06.283419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.283320 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl"] Nov 03 17:12:06.288684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.288662 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.308581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.308546 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl"] Nov 03 17:12:06.314217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.314185 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:12:06.314419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.314250 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:12:06.314573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.314544 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:12:06.314699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.314682 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:12:06.408204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.408161 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.408204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.408204 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.408616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.408599 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvtxp\" (UniqueName: \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.509966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.509924 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.509966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.509969 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.510396 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.510373 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvtxp\" (UniqueName: \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.510685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.510664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.512797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.512774 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.535722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.535635 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvtxp\" (UniqueName: \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\") pod \"maas-api-revoke-sub-daad1b09-2sfdl\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.610154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.610115 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:06.801862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.801758 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06424e2-36e1-4824-a50c-293db164710f" path="/var/lib/kubelet/pods/e06424e2-36e1-4824-a50c-293db164710f/volumes" Nov 03 17:12:06.925248 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:12:06.925214 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2427a2_7d72_4075_abe8_eff3d24d54fc.slice/crio-4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b WatchSource:0}: Error finding container 4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b: Status 404 returned error can't find the container with id 4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b Nov 03 17:12:06.928633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:06.928600 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl"] Nov 03 17:12:07.006979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:07.006931 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" event={"ID":"0d2427a2-7d72-4075-abe8-eff3d24d54fc","Type":"ContainerStarted","Data":"4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b"} Nov 03 17:12:08.017795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:08.017760 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" event={"ID":"0d2427a2-7d72-4075-abe8-eff3d24d54fc","Type":"ContainerStarted","Data":"480f25f2719cf0aff4afe77a5e683b84a31ade00a60a3eafe40b4f7cf3a7ab3d"} Nov 03 17:12:08.063616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:08.063549 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" podStartSLOduration=2.063531716 podStartE2EDuration="2.063531716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:08.06074025 +0000 UTC m=+3423.393953821" watchObservedRunningTime="2026-11-03 17:12:08.063531716 +0000 UTC m=+3423.396745285" Nov 03 17:12:09.050046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:09.050012 6001 generic.go:356] "Generic (PLEG): container finished" podID="0d2427a2-7d72-4075-abe8-eff3d24d54fc" containerID="480f25f2719cf0aff4afe77a5e683b84a31ade00a60a3eafe40b4f7cf3a7ab3d" exitCode=0 Nov 03 17:12:09.050496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:09.050094 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" event={"ID":"0d2427a2-7d72-4075-abe8-eff3d24d54fc","Type":"ContainerDied","Data":"480f25f2719cf0aff4afe77a5e683b84a31ade00a60a3eafe40b4f7cf3a7ab3d"} Nov 03 17:12:10.262837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.262804 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:10.376067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.375986 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\") pod \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " Nov 03 17:12:10.376226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.376139 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\" (UniqueName: \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\") pod \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " Nov 03 17:12:10.376273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.376240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\") pod \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\" (UID: \"0d2427a2-7d72-4075-abe8-eff3d24d54fc\") " Nov 03 17:12:10.376426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.376404 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca" pod "0d2427a2-7d72-4075-abe8-eff3d24d54fc" (UID: "0d2427a2-7d72-4075-abe8-eff3d24d54fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:10.377019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.376998 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:10.379131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.379108 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca" pod "0d2427a2-7d72-4075-abe8-eff3d24d54fc" (UID: "0d2427a2-7d72-4075-abe8-eff3d24d54fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:12:10.379647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.379624 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp" pod "0d2427a2-7d72-4075-abe8-eff3d24d54fc" (UID: "0d2427a2-7d72-4075-abe8-eff3d24d54fc"). InnerVolumeSpecName "kube-api-access-nvtxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:10.478911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.478878 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvtxp\" (UniqueName: \"kubernetes.io/projected/0d2427a2-7d72-4075-abe8-eff3d24d54fc-kube-api-access-nvtxp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:10.478911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:10.478908 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d2427a2-7d72-4075-abe8-eff3d24d54fc-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:11.125821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.125785 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r"] Nov 03 17:12:11.357378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.357329 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" event={"ID":"0d2427a2-7d72-4075-abe8-eff3d24d54fc","Type":"ContainerDied","Data":"4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b"} Nov 03 17:12:11.358895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.358871 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4f5e779ae22aa998d606d76c61d049ae5e1786d44b62b13d143a99d0ccf63b" Nov 03 17:12:11.363412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.363386 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl" Nov 03 17:12:11.384074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.384022 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f25d2502-gsf6r"] Nov 03 17:12:11.509680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:11.509645 6001 scope.go:122] "RemoveContainer" containerID="c26c56663c6e5ab6c4f99a5014cd8b50e0d69dbf67dec30acf33a3d6f388f1b0" Nov 03 17:12:12.791966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:12.791927 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7d3bb9a-723b-418c-a670-4935c13e97d7" path="/var/lib/kubelet/pods/f7d3bb9a-723b-418c-a670-4935c13e97d7/volumes" Nov 03 17:12:16.000481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:16.000406 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:12:22.252458 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:12:22.252404 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:12:24.093378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:24.093338 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5"] Nov 03 17:12:24.111699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:24.111663 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ff10a03-ppnr5"] Nov 03 17:12:24.794649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:24.794608 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b217f56-85d4-4515-9977-b58bebae9ae4" path="/var/lib/kubelet/pods/2b217f56-85d4-4515-9977-b58bebae9ae4/volumes" Nov 03 17:12:34.044178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.044138 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:34.046307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.046275 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0d2427a2-7d72-4075-abe8-eff3d24d54fc" containerName="revoke-keys" Nov 03 17:12:34.046307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.046305 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2427a2-7d72-4075-abe8-eff3d24d54fc" containerName="revoke-keys" Nov 03 17:12:34.046613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.046600 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2427a2-7d72-4075-abe8-eff3d24d54fc" containerName="revoke-keys" Nov 03 17:12:34.059784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.058253 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:34.059784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.058365 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.075063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.075031 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-f59fk\"" Nov 03 17:12:34.075063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.075050 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5fa6e4888e6c\"" Nov 03 17:12:34.087841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.087814 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 17:12:34.087841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.087821 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 17:12:34.088042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.087904 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 17:12:34.088042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.087981 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 17:12:34.192411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.191057 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.192411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.191129 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.192411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.191284 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkgj\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.192411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.191553 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.308673 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkgj\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.308970 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.309154 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.309192 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.310349 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.314521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.311328 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.351021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.350358 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.353059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.353023 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkgj\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\") pod \"payload-processing-external-model-8657c87f48-hnmmm\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.382104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.381785 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:34.817569 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:12:34.817534 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode032a0e3_fa2e_48fa_94f2_5a5dd9a49071.slice/crio-73ca68b1000524a2b4f5c469228b45edb97c58071e10386017995e2f992441a9 WatchSource:0}: Error finding container 73ca68b1000524a2b4f5c469228b45edb97c58071e10386017995e2f992441a9: Status 404 returned error can't find the container with id 73ca68b1000524a2b4f5c469228b45edb97c58071e10386017995e2f992441a9 Nov 03 17:12:34.820095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:34.820065 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:35.816966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:35.816927 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" event={"ID":"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071","Type":"ContainerStarted","Data":"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e"} Nov 03 17:12:35.819025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:35.817577 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" event={"ID":"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071","Type":"ContainerStarted","Data":"73ca68b1000524a2b4f5c469228b45edb97c58071e10386017995e2f992441a9"} Nov 03 17:12:35.819025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:35.818395 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:35.865631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:35.865574 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" podStartSLOduration=2.8655580240000003 podStartE2EDuration="2.865558024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:35.862670692 +0000 UTC m=+3451.195884262" watchObservedRunningTime="2026-11-03 17:12:35.865558024 +0000 UTC m=+3451.198771592" Nov 03 17:12:38.418171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:38.418137 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:42.891915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:42.891876 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:12:43.148247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.147770 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" containerID="cri-o://053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f" gracePeriod=30 Nov 03 17:12:43.434981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.434868 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" podUID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" containerName="payload-processing" containerID="cri-o://47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e" gracePeriod=25 Nov 03 17:12:43.776547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.776512 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:43.898804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.898760 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\") pod \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " Nov 03 17:12:43.899316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.898924 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\") pod \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " Nov 03 17:12:43.899316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.899117 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\") pod \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " Nov 03 17:12:43.899316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.899263 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\") pod \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\" (UID: \"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071\") " Nov 03 17:12:43.899521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.899390 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume" pod "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" (UID: "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:43.899686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.899662 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume" pod "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" (UID: "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:43.900300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.900280 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:43.900402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.900302 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:43.901785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.901756 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj" pod "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" (UID: "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071"). InnerVolumeSpecName "kube-api-access-dxkgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:43.901785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:43.901786 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials" pod "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" (UID: "e032a0e3-fa2e-48fa-94f2-5a5dd9a49071"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:44.002242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.002207 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkgj\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-kube-api-access-dxkgj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:44.002242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.002241 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:44.274526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.274382 6001 generic.go:356] "Generic (PLEG): container finished" podID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" containerID="47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e" exitCode=0 Nov 03 17:12:44.274747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.274696 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" event={"ID":"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071","Type":"ContainerDied","Data":"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e"} Nov 03 17:12:44.274826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.274758 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" event={"ID":"e032a0e3-fa2e-48fa-94f2-5a5dd9a49071","Type":"ContainerDied","Data":"73ca68b1000524a2b4f5c469228b45edb97c58071e10386017995e2f992441a9"} Nov 03 17:12:44.274826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.274788 6001 scope.go:122] "RemoveContainer" containerID="47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e" Nov 03 17:12:44.293475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.274978 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm" Nov 03 17:12:44.350602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.350564 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:12:44.352389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.352347 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" containerName="payload-processing" Nov 03 17:12:44.352742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.352718 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" containerName="payload-processing" Nov 03 17:12:44.353194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.353172 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" containerName="payload-processing" Nov 03 17:12:44.356771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.356695 6001 scope.go:122] "RemoveContainer" containerID="47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e" Nov 03 17:12:44.357255 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:12:44.357224 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e\": container with ID starting with 47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e not found: ID does not exist" containerID="47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e" Nov 03 17:12:44.357375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.357266 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e"} err="failed to get container status \"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e\": rpc error: code = NotFound desc = could not find container \"47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e\": container with ID starting with 47f0b3349b6b5bb83d276ba3ceaa1df04cc2cf370ccda299fc2c9492b116fd8e not found: ID does not exist" Nov 03 17:12:44.360702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.359993 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.381509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.381480 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"openshift-service-ca.crt\"" Nov 03 17:12:44.381690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.381621 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"routing-overlay\"" Nov 03 17:12:44.386049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.386021 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:12:44.388769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.388744 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"kube-root-ca.crt\"" Nov 03 17:12:44.388926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.388782 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-22a5b1e7c2da\"" Nov 03 17:12:44.388926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.388784 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e0e63d\"" Nov 03 17:12:44.390434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.390415 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-e2e--2172597b-dockercfg-xrxj5\"" Nov 03 17:12:44.411346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.411302 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:44.428145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.428101 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.428328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.428162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.428328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.428221 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.428602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.428568 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nsxx\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.429145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.429123 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-8657c87f48-hnmmm"] Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.533817 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.533890 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.533950 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.534219 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nsxx\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.548296 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.549045 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.551777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.549777 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.599469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.599272 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nsxx\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.678469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.677777 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:44.823276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:44.820766 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e032a0e3-fa2e-48fa-94f2-5a5dd9a49071" path="/var/lib/kubelet/pods/e032a0e3-fa2e-48fa-94f2-5a5dd9a49071/volumes" Nov 03 17:12:45.111951 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:12:45.111916 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af3eb48_c713_485a_bb3f_52d9555f89c5.slice/crio-fd622c6115c63dc6d05f746eba8dacd8468a79cc79bd3e81fc236ebf4adba2bd WatchSource:0}: Error finding container fd622c6115c63dc6d05f746eba8dacd8468a79cc79bd3e81fc236ebf4adba2bd: Status 404 returned error can't find the container with id fd622c6115c63dc6d05f746eba8dacd8468a79cc79bd3e81fc236ebf4adba2bd Nov 03 17:12:45.114078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:45.114047 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:12:45.442545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:45.440768 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" event={"ID":"2af3eb48-c713-485a-bb3f-52d9555f89c5","Type":"ContainerStarted","Data":"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b"} Nov 03 17:12:45.442545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:45.440851 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" event={"ID":"2af3eb48-c713-485a-bb3f-52d9555f89c5","Type":"ContainerStarted","Data":"fd622c6115c63dc6d05f746eba8dacd8468a79cc79bd3e81fc236ebf4adba2bd"} Nov 03 17:12:45.449548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:45.446212 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:45.500992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:45.500930 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" podStartSLOduration=1.5009144220000001 podStartE2EDuration="1.500914422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:45.497421765 +0000 UTC m=+3460.830635348" watchObservedRunningTime="2026-11-03 17:12:45.500914422 +0000 UTC m=+3460.834128059" Nov 03 17:12:48.442333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.442302 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:12:48.479285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.479242 6001 generic.go:356] "Generic (PLEG): container finished" podID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerID="053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f" exitCode=0 Nov 03 17:12:48.479509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.479331 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" Nov 03 17:12:48.479509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.479362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" event={"ID":"b2a5f7f1-2908-432e-9c29-dd221c55217c","Type":"ContainerDied","Data":"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f"} Nov 03 17:12:48.479509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.479429 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl" event={"ID":"b2a5f7f1-2908-432e-9c29-dd221c55217c","Type":"ContainerDied","Data":"98ce266b49d6be09c40d6bd04ddc49fba518010124630019a6ea87affb872caa"} Nov 03 17:12:48.479509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.479476 6001 scope.go:122] "RemoveContainer" containerID="053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f" Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559186 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559369 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559430 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559725 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559804 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559839 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559882 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.559953 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.565791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.560080 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\") pod \"b2a5f7f1-2908-432e-9c29-dd221c55217c\" (UID: \"b2a5f7f1-2908-432e-9c29-dd221c55217c\") " Nov 03 17:12:48.580742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.577438 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:12:48.580742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.577956 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:48.580742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.578208 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:12:48.580742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.580402 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:12:48.588875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.584879 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.588875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.584909 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.588875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.584924 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a5f7f1-2908-432e-9c29-dd221c55217c-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.588875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.584964 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.588875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.585230 6001 scope.go:122] "RemoveContainer" containerID="053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f" Nov 03 17:12:48.617969 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:12:48.586944 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f\": container with ID starting with 053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f not found: ID does not exist" containerID="053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f" Nov 03 17:12:48.618193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.618160 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f"} err="failed to get container status \"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f\": rpc error: code = NotFound desc = could not find container \"053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f\": container with ID starting with 053ece2246e1769b1f3ab2c91d046df150be6e334300781d6885b228cb5c031f not found: ID does not exist" Nov 03 17:12:48.618284 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.588352 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:48.618358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.588656 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:12:48.618429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.614653 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:12:48.618524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.616115 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "kube-api-access-hsfkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:48.620779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.620749 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy" pod "b2a5f7f1-2908-432e-9c29-dd221c55217c" (UID: "b2a5f7f1-2908-432e-9c29-dd221c55217c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:12:48.686391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.686297 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.686391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.686333 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.686391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.686343 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.686391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.686353 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsfkd\" (UniqueName: \"kubernetes.io/projected/b2a5f7f1-2908-432e-9c29-dd221c55217c-kube-api-access-hsfkd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.686391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.686364 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a5f7f1-2908-432e-9c29-dd221c55217c-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:12:48.833874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.833834 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:12:48.846682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:48.846644 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-c9wwl"] Nov 03 17:12:50.793940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:50.793897 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" path="/var/lib/kubelet/pods/b2a5f7f1-2908-432e-9c29-dd221c55217c/volumes" Nov 03 17:12:52.251990 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:12:52.251951 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:12:57.138185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.138145 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:12:57.139327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.139304 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" Nov 03 17:12:57.139407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.139330 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" Nov 03 17:12:57.139543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.139531 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a5f7f1-2908-432e-9c29-dd221c55217c" containerName="istio-proxy" Nov 03 17:12:57.145645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.145622 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.164049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.164017 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:12:57.170682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.170655 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-f6a669918c59\"" Nov 03 17:12:57.178239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.178185 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.178505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.178401 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsls\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.178505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.178469 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.178620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.178569 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.312470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.311697 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.312470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.311945 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsls\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.312470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.311994 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.312470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.312108 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.315470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.313273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.315470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.314007 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.329583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.322242 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.347482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.345360 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsls\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.464340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.464248 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:57.472317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.472291 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:12:57.790543 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:12:57.790512 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532c40e8_cea0_4c1c_81e1_6fa228e01245.slice/crio-728e44c791e4c54032ff45ec9c711c10bca32a625b27deb8e7505ba20a940213 WatchSource:0}: Error finding container 728e44c791e4c54032ff45ec9c711c10bca32a625b27deb8e7505ba20a940213: Status 404 returned error can't find the container with id 728e44c791e4c54032ff45ec9c711c10bca32a625b27deb8e7505ba20a940213 Nov 03 17:12:57.792875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:57.792848 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:12:58.018991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.018954 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:12:58.024419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.024397 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.043975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.043905 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"openshift-service-ca.crt\"" Nov 03 17:12:58.044143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.044062 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-da409b\"" Nov 03 17:12:58.048167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.048138 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:12:58.053955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.053877 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"routing-overlay\"" Nov 03 17:12:58.053955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.053927 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"kube-root-ca.crt\"" Nov 03 17:12:58.054133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.053958 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"agc-cred-dea1e9ecdfa6\"" Nov 03 17:12:58.054304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.054288 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-e2e--f0537159-dockercfg-vd526\"" Nov 03 17:12:58.054344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.054325 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"agc-cred-464294770e39\"" Nov 03 17:12:58.126809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.126771 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.126973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.126913 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.127036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.127008 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt9k2\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.127127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.127107 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.228665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.228627 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt9k2\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.229190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.228730 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.229190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.228958 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.229190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.229086 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.229736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.229706 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.229905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.229715 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.232178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.232155 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.253608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.253577 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt9k2\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.340340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.340257 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:58.655318 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:12:58.655287 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdc070b_80b5_49b5_adff_be91808c1761.slice/crio-b448d61f2a442234d059addad435bc6401da6d7ff25c8b1b70b266ce1ec91892 WatchSource:0}: Error finding container b448d61f2a442234d059addad435bc6401da6d7ff25c8b1b70b266ce1ec91892: Status 404 returned error can't find the container with id b448d61f2a442234d059addad435bc6401da6d7ff25c8b1b70b266ce1ec91892 Nov 03 17:12:58.655493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.655382 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:12:58.695124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.695090 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" event={"ID":"fbdc070b-80b5-49b5-adff-be91808c1761","Type":"ContainerStarted","Data":"b448d61f2a442234d059addad435bc6401da6d7ff25c8b1b70b266ce1ec91892"} Nov 03 17:12:58.697225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.697192 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" event={"ID":"532c40e8-cea0-4c1c-81e1-6fa228e01245","Type":"ContainerStarted","Data":"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd"} Nov 03 17:12:58.697369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.697248 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" event={"ID":"532c40e8-cea0-4c1c-81e1-6fa228e01245","Type":"ContainerStarted","Data":"728e44c791e4c54032ff45ec9c711c10bca32a625b27deb8e7505ba20a940213"} Nov 03 17:12:58.697369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.697276 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:12:58.746854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:58.746804 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" podStartSLOduration=1.746789194 podStartE2EDuration="1.746789194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:58.741583196 +0000 UTC m=+3474.074796764" watchObservedRunningTime="2026-11-03 17:12:58.746789194 +0000 UTC m=+3474.080002764" Nov 03 17:12:59.708017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:59.707968 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" event={"ID":"fbdc070b-80b5-49b5-adff-be91808c1761","Type":"ContainerStarted","Data":"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a"} Nov 03 17:12:59.708554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:59.708046 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:12:59.757103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:12:59.757053 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" podStartSLOduration=2.757038928 podStartE2EDuration="2.757038928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:59.752513002 +0000 UTC m=+3475.085726572" watchObservedRunningTime="2026-11-03 17:12:59.757038928 +0000 UTC m=+3475.090252508" Nov 03 17:13:09.712242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:09.711539 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:13:09.799610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:09.798264 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:13:10.457716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.457677 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt"] Nov 03 17:13:10.463608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.463580 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.483083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.483049 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt"] Nov 03 17:13:10.489569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.489484 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:13:10.489569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.489509 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:13:10.489569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.489539 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:13:10.491851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.491832 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:13:10.655961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.655924 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.656186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.655979 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274d7\" (UniqueName: \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.656372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.656348 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.746463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.743687 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:13:10.758903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.758849 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.759176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.759064 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.759176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.759101 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274d7\" (UniqueName: \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.760523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.760476 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.763380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.763354 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:10.785155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:10.785124 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274d7\" (UniqueName: \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\") pod \"maas-api-revoke-sub-5bbce2f2-4m6mt\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:11.081097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.081005 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:11.411868 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:13:11.411837 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe05aedd_b9d8_4def_8979_66031269a3bf.slice/crio-adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0 WatchSource:0}: Error finding container adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0: Status 404 returned error can't find the container with id adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0 Nov 03 17:13:11.413253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.413228 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt"] Nov 03 17:13:11.550061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.550028 6001 scope.go:122] "RemoveContainer" containerID="51edd7a28a96242121a2eec0d032a95af5c3145f2d0984124276a60f09a386ca" Nov 03 17:13:11.562023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.562001 6001 scope.go:122] "RemoveContainer" containerID="00d06d709478e12e69a63c2afe6c627d019cf47eb674e4b159d49378eb1355ff" Nov 03 17:13:11.574155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.574131 6001 scope.go:122] "RemoveContainer" containerID="9b99b1ad98cb051f74c8d4accbccbe1a78c5599f16b689f6299004fce5043c37" Nov 03 17:13:11.586191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.586164 6001 scope.go:122] "RemoveContainer" containerID="6a984215e753d6819562b7b4679fd34df75ee567f0ff2e755edb7c0287a43d1c" Nov 03 17:13:11.862359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:11.862328 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" event={"ID":"fe05aedd-b9d8-4def-8979-66031269a3bf","Type":"ContainerStarted","Data":"adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0"} Nov 03 17:13:12.873307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:12.873268 6001 generic.go:356] "Generic (PLEG): container finished" podID="fe05aedd-b9d8-4def-8979-66031269a3bf" containerID="8b2fbc8a5416955669bc14b55ed56770a151d0ca60c9d36b153d391a85a2ea46" exitCode=0 Nov 03 17:13:12.873715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:12.873341 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" event={"ID":"fe05aedd-b9d8-4def-8979-66031269a3bf","Type":"ContainerDied","Data":"8b2fbc8a5416955669bc14b55ed56770a151d0ca60c9d36b153d391a85a2ea46"} Nov 03 17:13:14.115139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.115106 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:14.223050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.223020 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\") pod \"fe05aedd-b9d8-4def-8979-66031269a3bf\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " Nov 03 17:13:14.223225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.223110 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\") pod \"fe05aedd-b9d8-4def-8979-66031269a3bf\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " Nov 03 17:13:14.223269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.223233 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\" (UniqueName: \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\") pod \"fe05aedd-b9d8-4def-8979-66031269a3bf\" (UID: \"fe05aedd-b9d8-4def-8979-66031269a3bf\") " Nov 03 17:13:14.223550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.223517 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca" pod "fe05aedd-b9d8-4def-8979-66031269a3bf" (UID: "fe05aedd-b9d8-4def-8979-66031269a3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:14.223966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.223945 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:14.225469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.225423 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca" pod "fe05aedd-b9d8-4def-8979-66031269a3bf" (UID: "fe05aedd-b9d8-4def-8979-66031269a3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:14.225568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.225518 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7" pod "fe05aedd-b9d8-4def-8979-66031269a3bf" (UID: "fe05aedd-b9d8-4def-8979-66031269a3bf"). InnerVolumeSpecName "kube-api-access-274d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:14.325758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.325718 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274d7\" (UniqueName: \"kubernetes.io/projected/fe05aedd-b9d8-4def-8979-66031269a3bf-kube-api-access-274d7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:14.325758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.325754 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe05aedd-b9d8-4def-8979-66031269a3bf-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:14.798776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.798735 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" podUID="2af3eb48-c713-485a-bb3f-52d9555f89c5" containerName="payload-processing" containerID="cri-o://159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b" gracePeriod=25 Nov 03 17:13:14.898095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.898063 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" Nov 03 17:13:14.898269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.898085 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt" event={"ID":"fe05aedd-b9d8-4def-8979-66031269a3bf","Type":"ContainerDied","Data":"adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0"} Nov 03 17:13:14.898269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:14.898140 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc789a611c70627716ae37ba4abb6c73721471989cd4fda3ac538d12bc867c0" Nov 03 17:13:15.074984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.074954 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:13:15.140048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.140014 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\") pod \"2af3eb48-c713-485a-bb3f-52d9555f89c5\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " Nov 03 17:13:15.140555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.140136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\") pod \"2af3eb48-c713-485a-bb3f-52d9555f89c5\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " Nov 03 17:13:15.140555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.140181 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\") pod \"2af3eb48-c713-485a-bb3f-52d9555f89c5\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " Nov 03 17:13:15.140555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.140537 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume" pod "2af3eb48-c713-485a-bb3f-52d9555f89c5" (UID: "2af3eb48-c713-485a-bb3f-52d9555f89c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:15.140769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.140620 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\") pod \"2af3eb48-c713-485a-bb3f-52d9555f89c5\" (UID: \"2af3eb48-c713-485a-bb3f-52d9555f89c5\") " Nov 03 17:13:15.141525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.141493 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume" pod "2af3eb48-c713-485a-bb3f-52d9555f89c5" (UID: "2af3eb48-c713-485a-bb3f-52d9555f89c5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:15.141737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.141712 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:15.141815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.141745 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2af3eb48-c713-485a-bb3f-52d9555f89c5-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:15.142799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.142769 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx" pod "2af3eb48-c713-485a-bb3f-52d9555f89c5" (UID: "2af3eb48-c713-485a-bb3f-52d9555f89c5"). InnerVolumeSpecName "kube-api-access-2nsxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:15.142964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.142831 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials" pod "2af3eb48-c713-485a-bb3f-52d9555f89c5" (UID: "2af3eb48-c713-485a-bb3f-52d9555f89c5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:15.243587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.243550 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nsxx\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-kube-api-access-2nsxx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:15.243587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.243582 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2af3eb48-c713-485a-bb3f-52d9555f89c5-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:15.908620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.908583 6001 generic.go:356] "Generic (PLEG): container finished" podID="2af3eb48-c713-485a-bb3f-52d9555f89c5" containerID="159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b" exitCode=0 Nov 03 17:13:15.908833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.908636 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" Nov 03 17:13:15.908833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.908673 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" event={"ID":"2af3eb48-c713-485a-bb3f-52d9555f89c5","Type":"ContainerDied","Data":"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b"} Nov 03 17:13:15.908833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.908734 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6" event={"ID":"2af3eb48-c713-485a-bb3f-52d9555f89c5","Type":"ContainerDied","Data":"fd622c6115c63dc6d05f746eba8dacd8468a79cc79bd3e81fc236ebf4adba2bd"} Nov 03 17:13:15.908833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.908761 6001 scope.go:122] "RemoveContainer" containerID="159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b" Nov 03 17:13:15.923830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.923675 6001 scope.go:122] "RemoveContainer" containerID="159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b" Nov 03 17:13:15.924070 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:13:15.924041 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b\": container with ID starting with 159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b not found: ID does not exist" containerID="159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b" Nov 03 17:13:15.924159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.924083 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b"} err="failed to get container status \"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b\": rpc error: code = NotFound desc = could not find container \"159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b\": container with ID starting with 159131a4cf8a310f4eb7a25597900b5147d4f35331cee56084d020bb17e50b2b not found: ID does not exist" Nov 03 17:13:15.966656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.966617 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:13:15.979704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:15.979671 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58ck2tl6"] Nov 03 17:13:16.794355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:16.794319 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2af3eb48-c713-485a-bb3f-52d9555f89c5" path="/var/lib/kubelet/pods/2af3eb48-c713-485a-bb3f-52d9555f89c5/volumes" Nov 03 17:13:21.297124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.297084 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz"] Nov 03 17:13:21.298075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298056 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fe05aedd-b9d8-4def-8979-66031269a3bf" containerName="revoke-keys" Nov 03 17:13:21.298160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298077 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe05aedd-b9d8-4def-8979-66031269a3bf" containerName="revoke-keys" Nov 03 17:13:21.298160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298127 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2af3eb48-c713-485a-bb3f-52d9555f89c5" containerName="payload-processing" Nov 03 17:13:21.298160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298132 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2af3eb48-c713-485a-bb3f-52d9555f89c5" containerName="payload-processing" Nov 03 17:13:21.298284 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298274 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2af3eb48-c713-485a-bb3f-52d9555f89c5" containerName="payload-processing" Nov 03 17:13:21.298323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.298290 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe05aedd-b9d8-4def-8979-66031269a3bf" containerName="revoke-keys" Nov 03 17:13:21.303910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.303880 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.324637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.324601 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz"] Nov 03 17:13:21.328579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.328548 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:13:21.328579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.328675 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:13:21.328579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.328750 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:13:21.332178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.332139 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:13:21.457023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.456981 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgv6s\" (UniqueName: \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.457262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.457137 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.457346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.457267 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.558924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.558827 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgv6s\" (UniqueName: \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.559100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.558952 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.559100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.559052 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.559766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.559740 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.561439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.561414 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.583950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.583918 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgv6s\" (UniqueName: \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\") pod \"maas-api-revoke-sub-d4434452-z2bbz\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.627366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.627321 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:21.946360 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:13:21.946329 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e65447_f0ef_45f2_8b98_e0d2790e4fcc.slice/crio-f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b WatchSource:0}: Error finding container f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b: Status 404 returned error can't find the container with id f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b Nov 03 17:13:21.949189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.949156 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz"] Nov 03 17:13:21.986987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:21.986933 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" event={"ID":"87e65447-f0ef-45f2-8b98-e0d2790e4fcc","Type":"ContainerStarted","Data":"f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b"} Nov 03 17:13:22.252748 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:13:22.252703 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:13:22.998988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:22.998955 6001 generic.go:356] "Generic (PLEG): container finished" podID="87e65447-f0ef-45f2-8b98-e0d2790e4fcc" containerID="d595202e369f819e5b35b3c94220895c7b15af675f4de390309ff347c5dd5b1b" exitCode=0 Nov 03 17:13:22.999411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:22.999035 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" event={"ID":"87e65447-f0ef-45f2-8b98-e0d2790e4fcc","Type":"ContainerDied","Data":"d595202e369f819e5b35b3c94220895c7b15af675f4de390309ff347c5dd5b1b"} Nov 03 17:13:24.206740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.206713 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:24.334993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.334888 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\") pod \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " Nov 03 17:13:24.335173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.335041 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\" (UniqueName: \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\") pod \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " Nov 03 17:13:24.335173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.335103 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\") pod \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\" (UID: \"87e65447-f0ef-45f2-8b98-e0d2790e4fcc\") " Nov 03 17:13:24.335608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.335581 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca" pod "87e65447-f0ef-45f2-8b98-e0d2790e4fcc" (UID: "87e65447-f0ef-45f2-8b98-e0d2790e4fcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:24.335943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.335926 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:24.337097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.337074 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca" pod "87e65447-f0ef-45f2-8b98-e0d2790e4fcc" (UID: "87e65447-f0ef-45f2-8b98-e0d2790e4fcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:24.337222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.337200 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s" pod "87e65447-f0ef-45f2-8b98-e0d2790e4fcc" (UID: "87e65447-f0ef-45f2-8b98-e0d2790e4fcc"). InnerVolumeSpecName "kube-api-access-cgv6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:24.437737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.437697 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgv6s\" (UniqueName: \"kubernetes.io/projected/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-kube-api-access-cgv6s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:24.437737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:24.437735 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e65447-f0ef-45f2-8b98-e0d2790e4fcc-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:25.020036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:25.019999 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" Nov 03 17:13:25.020036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:25.020025 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz" event={"ID":"87e65447-f0ef-45f2-8b98-e0d2790e4fcc","Type":"ContainerDied","Data":"f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b"} Nov 03 17:13:25.020292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:25.020083 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ff0af7eded366f8b89a581ae4bafb73eade9eeaacd658ff0651f6b7323de2b" Nov 03 17:13:32.978696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:32.978662 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:32.979635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:32.979618 6001 cpu_manager.go:414] "containerMap: removing container" podUID="87e65447-f0ef-45f2-8b98-e0d2790e4fcc" containerName="revoke-keys" Nov 03 17:13:32.979635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:32.979636 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e65447-f0ef-45f2-8b98-e0d2790e4fcc" containerName="revoke-keys" Nov 03 17:13:32.979856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:32.979845 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e65447-f0ef-45f2-8b98-e0d2790e4fcc" containerName="revoke-keys" Nov 03 17:13:32.994531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:32.993327 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.007180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.007147 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:33.011329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.011300 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"agc-cred-edddadb39aba\"" Nov 03 17:13:33.011329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.011306 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"routing-overlay\"" Nov 03 17:13:33.011595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.011374 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-e2e--e145110a-dockercfg-lpr5l\"" Nov 03 17:13:33.011595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.011428 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"kube-root-ca.crt\"" Nov 03 17:13:33.023013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.022987 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"openshift-service-ca.crt\"" Nov 03 17:13:33.023175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.023034 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d6fcae\"" Nov 03 17:13:33.116589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.116550 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.116815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.116681 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.116968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.116836 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.117144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.117125 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sch6w\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.219342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.219297 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.219594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.219555 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sch6w\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.219683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.219632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.219683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.219681 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.220170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.220142 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.220277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.220263 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.222526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.222421 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.244324 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.244253 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sch6w\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.313926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.313875 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:33.629278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.629097 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:33.642831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:33.642801 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:34.227854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:34.224970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" event={"ID":"f789e8fa-137d-4cfd-8a8c-1122714fccb2","Type":"ContainerStarted","Data":"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51"} Nov 03 17:13:34.227854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:34.225042 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" event={"ID":"f789e8fa-137d-4cfd-8a8c-1122714fccb2","Type":"ContainerStarted","Data":"04fe906e81b4d1378fa39f5ff47889f350c2384451ea67ccf04680c08bcf8860"} Nov 03 17:13:34.227854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:34.225217 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:34.333972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:34.333908 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" podStartSLOduration=2.333884412 podStartE2EDuration="2.333884412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:34.291919001 +0000 UTC m=+3509.625132588" watchObservedRunningTime="2026-11-03 17:13:34.333884412 +0000 UTC m=+3509.667097982" Nov 03 17:13:39.225737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.225687 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" podUID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" containerName="payload-processing" containerID="cri-o://1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51" gracePeriod=25 Nov 03 17:13:39.589607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.589577 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:39.655889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.655854 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\") pod \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " Nov 03 17:13:39.656109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.655917 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\") pod \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " Nov 03 17:13:39.656109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.655937 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\") pod \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " Nov 03 17:13:39.656109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.655960 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\") pod \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\" (UID: \"f789e8fa-137d-4cfd-8a8c-1122714fccb2\") " Nov 03 17:13:39.656558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.656528 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume" pod "f789e8fa-137d-4cfd-8a8c-1122714fccb2" (UID: "f789e8fa-137d-4cfd-8a8c-1122714fccb2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:39.656744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.656717 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume" pod "f789e8fa-137d-4cfd-8a8c-1122714fccb2" (UID: "f789e8fa-137d-4cfd-8a8c-1122714fccb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:39.657026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.657005 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.657122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.657032 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f789e8fa-137d-4cfd-8a8c-1122714fccb2-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.658611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.658580 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w" pod "f789e8fa-137d-4cfd-8a8c-1122714fccb2" (UID: "f789e8fa-137d-4cfd-8a8c-1122714fccb2"). InnerVolumeSpecName "kube-api-access-sch6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:39.658888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.658864 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials" pod "f789e8fa-137d-4cfd-8a8c-1122714fccb2" (UID: "f789e8fa-137d-4cfd-8a8c-1122714fccb2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:39.759001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.758951 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.759001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:39.758994 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sch6w\" (UniqueName: \"kubernetes.io/projected/f789e8fa-137d-4cfd-8a8c-1122714fccb2-kube-api-access-sch6w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:40.282406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.282371 6001 generic.go:356] "Generic (PLEG): container finished" podID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" containerID="1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51" exitCode=0 Nov 03 17:13:40.282921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.282424 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" Nov 03 17:13:40.282921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.282491 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" event={"ID":"f789e8fa-137d-4cfd-8a8c-1122714fccb2","Type":"ContainerDied","Data":"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51"} Nov 03 17:13:40.282921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.282553 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph" event={"ID":"f789e8fa-137d-4cfd-8a8c-1122714fccb2","Type":"ContainerDied","Data":"04fe906e81b4d1378fa39f5ff47889f350c2384451ea67ccf04680c08bcf8860"} Nov 03 17:13:40.282921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.282577 6001 scope.go:122] "RemoveContainer" containerID="1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51" Nov 03 17:13:40.295173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.295152 6001 scope.go:122] "RemoveContainer" containerID="1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51" Nov 03 17:13:40.295520 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:13:40.295493 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51\": container with ID starting with 1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51 not found: ID does not exist" containerID="1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51" Nov 03 17:13:40.295609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.295535 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51"} err="failed to get container status \"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51\": rpc error: code = NotFound desc = could not find container \"1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51\": container with ID starting with 1bfe9349a4fbabd7ebe65894229ade76e60cb4b7e3fddb2a40b39cfce4ac0c51 not found: ID does not exist" Nov 03 17:13:40.340233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.340195 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:40.352454 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.352411 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-5955wndph"] Nov 03 17:13:40.792978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:40.792931 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" path="/var/lib/kubelet/pods/f789e8fa-137d-4cfd-8a8c-1122714fccb2/volumes" Nov 03 17:13:41.066541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:41.065107 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t"] Nov 03 17:13:41.085549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:41.079331 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ee5f33d8-htn6t"] Nov 03 17:13:42.793127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:42.793077 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eab1e11-c447-418b-9423-ec5e532f4067" path="/var/lib/kubelet/pods/0eab1e11-c447-418b-9423-ec5e532f4067/volumes" Nov 03 17:13:52.037150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.037059 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7"] Nov 03 17:13:52.043211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.043178 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s"] Nov 03 17:13:52.044215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.044192 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" containerName="payload-processing" Nov 03 17:13:52.044215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.044213 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" containerName="payload-processing" Nov 03 17:13:52.044387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.044381 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f789e8fa-137d-4cfd-8a8c-1122714fccb2" containerName="payload-processing" Nov 03 17:13:52.050135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.050105 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86a2b42e-2pzm7"] Nov 03 17:13:52.050255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.050193 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.063788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.063757 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s"] Nov 03 17:13:52.065773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.065750 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:13:52.066036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.066015 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:13:52.066036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.066019 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:13:52.066192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.066057 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:13:52.107331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.107294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfzrc\" (UniqueName: \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.107591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.107364 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.107591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.107512 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.209823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.209770 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfzrc\" (UniqueName: \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.210099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.209869 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.210099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.209931 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.210586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.210563 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.212357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.212339 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.234423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.234387 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfzrc\" (UniqueName: \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\") pod \"maas-api-revoke-sub-f210497d-xg64s\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.315898 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:13:52.315781 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:13:52.367950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.367908 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:52.688497 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:13:52.687954 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70b7386_efc5_4549_bd6a_461ec3e24c21.slice/crio-bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46 WatchSource:0}: Error finding container bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46: Status 404 returned error can't find the container with id bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46 Nov 03 17:13:52.691240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.691209 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s"] Nov 03 17:13:52.793122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:52.793087 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2030228a-dd14-48d4-a041-38dc73aa9b10" path="/var/lib/kubelet/pods/2030228a-dd14-48d4-a041-38dc73aa9b10/volumes" Nov 03 17:13:53.532140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:53.532101 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" event={"ID":"b70b7386-efc5-4549-bd6a-461ec3e24c21","Type":"ContainerStarted","Data":"d6a9747fc0762aa5dfde0176d47615a971120a964a941703c57b7d8c0bb217be"} Nov 03 17:13:53.532621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:53.532160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" event={"ID":"b70b7386-efc5-4549-bd6a-461ec3e24c21","Type":"ContainerStarted","Data":"bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46"} Nov 03 17:13:53.579768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:53.579714 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" podStartSLOduration=1.579696836 podStartE2EDuration="1.579696836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:53.576385972 +0000 UTC m=+3528.909599543" watchObservedRunningTime="2026-11-03 17:13:53.579696836 +0000 UTC m=+3528.912910404" Nov 03 17:13:54.622746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:54.622705 6001 generic.go:356] "Generic (PLEG): container finished" podID="b70b7386-efc5-4549-bd6a-461ec3e24c21" containerID="d6a9747fc0762aa5dfde0176d47615a971120a964a941703c57b7d8c0bb217be" exitCode=0 Nov 03 17:13:54.623477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:54.623427 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" event={"ID":"b70b7386-efc5-4549-bd6a-461ec3e24c21","Type":"ContainerDied","Data":"d6a9747fc0762aa5dfde0176d47615a971120a964a941703c57b7d8c0bb217be"} Nov 03 17:13:55.353533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.353496 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:13:55.359094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.359066 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.380929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.380895 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"agc-cred-283408759ba7\"" Nov 03 17:13:55.380929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.380897 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"agc-cred-415f79a03b4e\"" Nov 03 17:13:55.381249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.381226 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-e2e--f8f5d575-dockercfg-zjb2n\"" Nov 03 17:13:55.394072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.394041 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"openshift-service-ca.crt\"" Nov 03 17:13:55.394326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.394199 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdbf0c\"" Nov 03 17:13:55.394326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.394308 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"kube-root-ca.crt\"" Nov 03 17:13:55.394549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.394248 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"routing-overlay\"" Nov 03 17:13:55.401458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.401415 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:13:55.474926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.474880 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbnbp\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.475128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.475017 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.475128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.475061 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.475230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.475170 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.576991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.576929 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbnbp\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.577203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.577056 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.577203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.577102 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.577343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.577209 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.577871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.577841 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.578009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.577992 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.579914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.579891 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.605415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.605341 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbnbp\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.675981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.675940 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:55.838697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.838665 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:55.983511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.983472 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\") pod \"b70b7386-efc5-4549-bd6a-461ec3e24c21\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " Nov 03 17:13:55.983708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.983617 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\" (UniqueName: \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\") pod \"b70b7386-efc5-4549-bd6a-461ec3e24c21\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " Nov 03 17:13:55.983708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.983691 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\") pod \"b70b7386-efc5-4549-bd6a-461ec3e24c21\" (UID: \"b70b7386-efc5-4549-bd6a-461ec3e24c21\") " Nov 03 17:13:55.984071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.984040 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca" pod "b70b7386-efc5-4549-bd6a-461ec3e24c21" (UID: "b70b7386-efc5-4549-bd6a-461ec3e24c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:55.984505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.984436 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:55.985755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.985732 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca" pod "b70b7386-efc5-4549-bd6a-461ec3e24c21" (UID: "b70b7386-efc5-4549-bd6a-461ec3e24c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:13:55.985901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:55.985882 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc" pod "b70b7386-efc5-4549-bd6a-461ec3e24c21" (UID: "b70b7386-efc5-4549-bd6a-461ec3e24c21"). InnerVolumeSpecName "kube-api-access-bfzrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:56.039073 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:13:56.039035 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25523fec_a8b2_4a74_b9e0_525b3d18e642.slice/crio-ca4276efb6c023e1a8cc8aa6a6d910d58a478990bd22c3f04e16ecb483cf8471 WatchSource:0}: Error finding container ca4276efb6c023e1a8cc8aa6a6d910d58a478990bd22c3f04e16ecb483cf8471: Status 404 returned error can't find the container with id ca4276efb6c023e1a8cc8aa6a6d910d58a478990bd22c3f04e16ecb483cf8471 Nov 03 17:13:56.040171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.040145 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:13:56.085923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.085893 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70b7386-efc5-4549-bd6a-461ec3e24c21-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:56.085923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.085923 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfzrc\" (UniqueName: \"kubernetes.io/projected/b70b7386-efc5-4549-bd6a-461ec3e24c21-kube-api-access-bfzrc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:13:56.645514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.645060 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" event={"ID":"25523fec-a8b2-4a74-b9e0-525b3d18e642","Type":"ContainerStarted","Data":"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2"} Nov 03 17:13:56.645514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.645155 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" event={"ID":"25523fec-a8b2-4a74-b9e0-525b3d18e642","Type":"ContainerStarted","Data":"ca4276efb6c023e1a8cc8aa6a6d910d58a478990bd22c3f04e16ecb483cf8471"} Nov 03 17:13:56.645514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.645410 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:13:56.647256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.647229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" event={"ID":"b70b7386-efc5-4549-bd6a-461ec3e24c21","Type":"ContainerDied","Data":"bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46"} Nov 03 17:13:56.647389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.647269 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd732bdec765e999bf7f752242aca8d337e3be62d38ba7a64ec717b6884cff46" Nov 03 17:13:56.647389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.647320 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s" Nov 03 17:13:56.693896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:13:56.693840 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" podStartSLOduration=1.693822341 podStartE2EDuration="1.693822341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:56.689840746 +0000 UTC m=+3532.023054319" watchObservedRunningTime="2026-11-03 17:13:56.693822341 +0000 UTC m=+3532.027035910" Nov 03 17:14:02.106369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.091734 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn"] Nov 03 17:14:02.129472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.124766 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b70b7386-efc5-4549-bd6a-461ec3e24c21" containerName="revoke-keys" Nov 03 17:14:02.129472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.124796 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70b7386-efc5-4549-bd6a-461ec3e24c21" containerName="revoke-keys" Nov 03 17:14:02.145464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.138222 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70b7386-efc5-4549-bd6a-461ec3e24c21" containerName="revoke-keys" Nov 03 17:14:02.265229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.261206 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn"] Nov 03 17:14:02.265229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.261422 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.290680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.290641 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:14:02.290680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.290667 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:14:02.290680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.290657 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:14:02.290946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.290704 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:14:02.360527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.357129 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.360527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.357259 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.360527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.357393 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkwc\" (UniqueName: \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.476464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.476269 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.476464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.476380 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.476759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.476527 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkwc\" (UniqueName: \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.488474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.477789 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.491750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.491716 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.517338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.517301 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkwc\" (UniqueName: \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\") pod \"maas-api-revoke-sub-eaff300f-72qvn\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.582637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.582591 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:02.728093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.727624 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:14:02.731253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.730754 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:14:02.928019 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:02.927986 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73810920_f5ae_4a5b_bebc_77df1898b031.slice/crio-144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df WatchSource:0}: Error finding container 144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df: Status 404 returned error can't find the container with id 144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df Nov 03 17:14:02.931263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:02.931237 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn"] Nov 03 17:14:03.727392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:03.727356 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" event={"ID":"73810920-f5ae-4a5b-bebc-77df1898b031","Type":"ContainerStarted","Data":"7b99b304f650bdfe52a5e6de6615b8251a30acbdc80af39c149199b432d44d47"} Nov 03 17:14:03.727882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:03.727419 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" event={"ID":"73810920-f5ae-4a5b-bebc-77df1898b031","Type":"ContainerStarted","Data":"144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df"} Nov 03 17:14:03.772352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:03.772274 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" podStartSLOduration=1.77225396 podStartE2EDuration="1.77225396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:03.772047881 +0000 UTC m=+3539.105261452" watchObservedRunningTime="2026-11-03 17:14:03.77225396 +0000 UTC m=+3539.105467530" Nov 03 17:14:04.358566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.358526 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:14:04.534197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.534108 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7"] Nov 03 17:14:04.544716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.544680 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.549300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.549263 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7"] Nov 03 17:14:04.561154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.559493 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.561154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.559633 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b675z\" (UniqueName: \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.561154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.559673 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.661606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.661564 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b675z\" (UniqueName: \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.661606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.661615 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.661909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.661896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.662379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.662354 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.664605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.664579 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.693113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.693082 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b675z\" (UniqueName: \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\") pod \"maas-api-revoke-sub-e95d963b-c7nq7\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:04.740327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.740293 6001 generic.go:356] "Generic (PLEG): container finished" podID="73810920-f5ae-4a5b-bebc-77df1898b031" containerID="7b99b304f650bdfe52a5e6de6615b8251a30acbdc80af39c149199b432d44d47" exitCode=0 Nov 03 17:14:04.740753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.740376 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" event={"ID":"73810920-f5ae-4a5b-bebc-77df1898b031","Type":"ContainerDied","Data":"7b99b304f650bdfe52a5e6de6615b8251a30acbdc80af39c149199b432d44d47"} Nov 03 17:14:04.888528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:04.887798 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:05.135951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:05.135903 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7"] Nov 03 17:14:05.145434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:05.145350 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0e53b5e-scpm7"] Nov 03 17:14:05.435083 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:05.435051 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe4815e_29e6_48c8_a035_ef1c00bc571c.slice/crio-57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044 WatchSource:0}: Error finding container 57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044: Status 404 returned error can't find the container with id 57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044 Nov 03 17:14:05.438098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:05.438070 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7"] Nov 03 17:14:05.752088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:05.752049 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" event={"ID":"4fe4815e-29e6-48c8-a035-ef1c00bc571c","Type":"ContainerStarted","Data":"57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044"} Nov 03 17:14:05.931993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:05.931961 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:06.020592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.020496 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\") pod \"73810920-f5ae-4a5b-bebc-77df1898b031\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " Nov 03 17:14:06.021283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.021259 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\") pod \"73810920-f5ae-4a5b-bebc-77df1898b031\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " Nov 03 17:14:06.021669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.021590 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\" (UniqueName: \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\") pod \"73810920-f5ae-4a5b-bebc-77df1898b031\" (UID: \"73810920-f5ae-4a5b-bebc-77df1898b031\") " Nov 03 17:14:06.023411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.023371 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca" pod "73810920-f5ae-4a5b-bebc-77df1898b031" (UID: "73810920-f5ae-4a5b-bebc-77df1898b031"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:06.055112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.055053 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc" pod "73810920-f5ae-4a5b-bebc-77df1898b031" (UID: "73810920-f5ae-4a5b-bebc-77df1898b031"). InnerVolumeSpecName "kube-api-access-tgkwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:06.061679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.061067 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57"] Nov 03 17:14:06.064489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.064428 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca" pod "73810920-f5ae-4a5b-bebc-77df1898b031" (UID: "73810920-f5ae-4a5b-bebc-77df1898b031"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:06.069505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.069473 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8"] Nov 03 17:14:06.078862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.078791 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8775f97-fxz57"] Nov 03 17:14:06.087320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.087293 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6e6c0e8-6k6j8"] Nov 03 17:14:06.124306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.124273 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:06.124306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.124307 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73810920-f5ae-4a5b-bebc-77df1898b031-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:06.124613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.124323 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkwc\" (UniqueName: \"kubernetes.io/projected/73810920-f5ae-4a5b-bebc-77df1898b031-kube-api-access-tgkwc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:06.762430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.762397 6001 generic.go:356] "Generic (PLEG): container finished" podID="4fe4815e-29e6-48c8-a035-ef1c00bc571c" containerID="ed5e29a9ab1d0fa4c0d42648fb408bb3347eb0fb86e007b54e843df3a36e1301" exitCode=0 Nov 03 17:14:06.762892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.762492 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" event={"ID":"4fe4815e-29e6-48c8-a035-ef1c00bc571c","Type":"ContainerDied","Data":"ed5e29a9ab1d0fa4c0d42648fb408bb3347eb0fb86e007b54e843df3a36e1301"} Nov 03 17:14:06.764049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.764028 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" Nov 03 17:14:06.764181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.764063 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn" event={"ID":"73810920-f5ae-4a5b-bebc-77df1898b031","Type":"ContainerDied","Data":"144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df"} Nov 03 17:14:06.764181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.764108 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="144c87da06a04ad4f8c5c01943a0d01f1b079aafbd4093b8f59fd9f636a7c5df" Nov 03 17:14:06.793392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.793362 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20eb019b-4ab4-4bd0-9a86-48aaf137d755" path="/var/lib/kubelet/pods/20eb019b-4ab4-4bd0-9a86-48aaf137d755/volumes" Nov 03 17:14:06.793979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.793961 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ef1ad7-d9a4-402c-bbd7-eba49a40e39e" path="/var/lib/kubelet/pods/52ef1ad7-d9a4-402c-bbd7-eba49a40e39e/volumes" Nov 03 17:14:06.794424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:06.794407 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75245d93-f369-4d19-aadc-2c8e0522f721" path="/var/lib/kubelet/pods/75245d93-f369-4d19-aadc-2c8e0522f721/volumes" Nov 03 17:14:07.728227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:07.728182 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" podUID="25523fec-a8b2-4a74-b9e0-525b3d18e642" containerName="payload-processing" containerID="cri-o://b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2" gracePeriod=25 Nov 03 17:14:08.005496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.005404 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:08.063808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.063773 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\") pod \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " Nov 03 17:14:08.064055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.064035 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\") pod \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " Nov 03 17:14:08.064145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.064097 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\" (UniqueName: \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\") pod \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\" (UID: \"4fe4815e-29e6-48c8-a035-ef1c00bc571c\") " Nov 03 17:14:08.064714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.064686 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca" pod "4fe4815e-29e6-48c8-a035-ef1c00bc571c" (UID: "4fe4815e-29e6-48c8-a035-ef1c00bc571c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:08.064930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.064914 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.067254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.067224 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca" pod "4fe4815e-29e6-48c8-a035-ef1c00bc571c" (UID: "4fe4815e-29e6-48c8-a035-ef1c00bc571c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:08.069597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.069574 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z" pod "4fe4815e-29e6-48c8-a035-ef1c00bc571c" (UID: "4fe4815e-29e6-48c8-a035-ef1c00bc571c"). InnerVolumeSpecName "kube-api-access-b675z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:08.093336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.093312 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:14:08.165985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.165937 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\") pod \"25523fec-a8b2-4a74-b9e0-525b3d18e642\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " Nov 03 17:14:08.166194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.166101 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\") pod \"25523fec-a8b2-4a74-b9e0-525b3d18e642\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " Nov 03 17:14:08.166384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.166263 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\") pod \"25523fec-a8b2-4a74-b9e0-525b3d18e642\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " Nov 03 17:14:08.166496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.166388 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\") pod \"25523fec-a8b2-4a74-b9e0-525b3d18e642\" (UID: \"25523fec-a8b2-4a74-b9e0-525b3d18e642\") " Nov 03 17:14:08.166646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.166601 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume" pod "25523fec-a8b2-4a74-b9e0-525b3d18e642" (UID: "25523fec-a8b2-4a74-b9e0-525b3d18e642"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:08.167147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.167119 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume" pod "25523fec-a8b2-4a74-b9e0-525b3d18e642" (UID: "25523fec-a8b2-4a74-b9e0-525b3d18e642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:08.167328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.167309 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fe4815e-29e6-48c8-a035-ef1c00bc571c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.167402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.167334 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.167402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.167351 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25523fec-a8b2-4a74-b9e0-525b3d18e642-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.167402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.167365 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b675z\" (UniqueName: \"kubernetes.io/projected/4fe4815e-29e6-48c8-a035-ef1c00bc571c-kube-api-access-b675z\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.168666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.168642 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp" pod "25523fec-a8b2-4a74-b9e0-525b3d18e642" (UID: "25523fec-a8b2-4a74-b9e0-525b3d18e642"). InnerVolumeSpecName "kube-api-access-rbnbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:08.168961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.168937 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials" pod "25523fec-a8b2-4a74-b9e0-525b3d18e642" (UID: "25523fec-a8b2-4a74-b9e0-525b3d18e642"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:08.268739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.268653 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.268739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.268679 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbnbp\" (UniqueName: \"kubernetes.io/projected/25523fec-a8b2-4a74-b9e0-525b3d18e642-kube-api-access-rbnbp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:08.788739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.788706 6001 generic.go:356] "Generic (PLEG): container finished" podID="25523fec-a8b2-4a74-b9e0-525b3d18e642" containerID="b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2" exitCode=0 Nov 03 17:14:08.788936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.788806 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" Nov 03 17:14:08.790808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.790785 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" Nov 03 17:14:08.793317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.793289 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" event={"ID":"25523fec-a8b2-4a74-b9e0-525b3d18e642","Type":"ContainerDied","Data":"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2"} Nov 03 17:14:08.793523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.793337 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr" event={"ID":"25523fec-a8b2-4a74-b9e0-525b3d18e642","Type":"ContainerDied","Data":"ca4276efb6c023e1a8cc8aa6a6d910d58a478990bd22c3f04e16ecb483cf8471"} Nov 03 17:14:08.793523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.793355 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7" event={"ID":"4fe4815e-29e6-48c8-a035-ef1c00bc571c","Type":"ContainerDied","Data":"57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044"} Nov 03 17:14:08.793523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.793373 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57556325653c321b5146d4300b9ecca93222fab4ad6b174e073c883269bd9044" Nov 03 17:14:08.793523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.793388 6001 scope.go:122] "RemoveContainer" containerID="b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2" Nov 03 17:14:08.808049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.808021 6001 scope.go:122] "RemoveContainer" containerID="b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2" Nov 03 17:14:08.808380 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:08.808360 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2\": container with ID starting with b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2 not found: ID does not exist" containerID="b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2" Nov 03 17:14:08.808457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.808389 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2"} err="failed to get container status \"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2\": rpc error: code = NotFound desc = could not find container \"b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2\": container with ID starting with b83180a9fd69a459c05aed484693fab95609d54f2d81343191b442d17d5184f2 not found: ID does not exist" Nov 03 17:14:08.849982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.849945 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:14:08.862128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:08.862098 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-7745npzzr"] Nov 03 17:14:09.360146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.360070 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" podUID="a5f9da16-e423-4203-a8f7-e275b8b435eb" containerName="payload-processing" containerID="cri-o://79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083" gracePeriod=25 Nov 03 17:14:09.666960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.666931 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:14:09.692854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.692821 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\") pod \"a5f9da16-e423-4203-a8f7-e275b8b435eb\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " Nov 03 17:14:09.693034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.692915 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\") pod \"a5f9da16-e423-4203-a8f7-e275b8b435eb\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " Nov 03 17:14:09.693238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.693218 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\") pod \"a5f9da16-e423-4203-a8f7-e275b8b435eb\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " Nov 03 17:14:09.693286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.693272 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\") pod \"a5f9da16-e423-4203-a8f7-e275b8b435eb\" (UID: \"a5f9da16-e423-4203-a8f7-e275b8b435eb\") " Nov 03 17:14:09.693345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.693306 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume" pod "a5f9da16-e423-4203-a8f7-e275b8b435eb" (UID: "a5f9da16-e423-4203-a8f7-e275b8b435eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:09.693699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.693674 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume" pod "a5f9da16-e423-4203-a8f7-e275b8b435eb" (UID: "a5f9da16-e423-4203-a8f7-e275b8b435eb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:09.694149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.694130 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:09.694219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.694155 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5f9da16-e423-4203-a8f7-e275b8b435eb-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:09.695549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.695525 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials" pod "a5f9da16-e423-4203-a8f7-e275b8b435eb" (UID: "a5f9da16-e423-4203-a8f7-e275b8b435eb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:09.695644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.695566 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq" pod "a5f9da16-e423-4203-a8f7-e275b8b435eb" (UID: "a5f9da16-e423-4203-a8f7-e275b8b435eb"). InnerVolumeSpecName "kube-api-access-wm8cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:09.796425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.796387 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm8cq\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-kube-api-access-wm8cq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:09.796425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.796426 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5f9da16-e423-4203-a8f7-e275b8b435eb-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:09.803247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.803212 6001 generic.go:356] "Generic (PLEG): container finished" podID="a5f9da16-e423-4203-a8f7-e275b8b435eb" containerID="79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083" exitCode=0 Nov 03 17:14:09.803391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.803261 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" Nov 03 17:14:09.803391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.803297 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" event={"ID":"a5f9da16-e423-4203-a8f7-e275b8b435eb","Type":"ContainerDied","Data":"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083"} Nov 03 17:14:09.803391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.803350 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv" event={"ID":"a5f9da16-e423-4203-a8f7-e275b8b435eb","Type":"ContainerDied","Data":"71f32e1e4d1f3ee7b7901572d0f17b6bd997f6ab8fdd6261971bcddabacc9e9b"} Nov 03 17:14:09.803391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.803372 6001 scope.go:122] "RemoveContainer" containerID="79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083" Nov 03 17:14:09.815856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.815837 6001 scope.go:122] "RemoveContainer" containerID="79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083" Nov 03 17:14:09.816135 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:09.816117 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083\": container with ID starting with 79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083 not found: ID does not exist" containerID="79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083" Nov 03 17:14:09.816191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.816144 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083"} err="failed to get container status \"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083\": rpc error: code = NotFound desc = could not find container \"79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083\": container with ID starting with 79998b2baad805c4293a5639543313358f1c1cea5d0ea4d1829357f184a6f083 not found: ID does not exist" Nov 03 17:14:09.887418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.887333 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:14:09.892415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:09.892384 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6c5dlb8kv"] Nov 03 17:14:10.792170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:10.792133 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25523fec-a8b2-4a74-b9e0-525b3d18e642" path="/var/lib/kubelet/pods/25523fec-a8b2-4a74-b9e0-525b3d18e642/volumes" Nov 03 17:14:10.792785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:10.792768 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f9da16-e423-4203-a8f7-e275b8b435eb" path="/var/lib/kubelet/pods/a5f9da16-e423-4203-a8f7-e275b8b435eb/volumes" Nov 03 17:14:11.657209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:11.657176 6001 scope.go:122] "RemoveContainer" containerID="e35eba4d65147c00473df80b06db4e6cfab1d5196b215991dc25d0130403e237" Nov 03 17:14:12.851416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.851377 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk"] Nov 03 17:14:12.852235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852219 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a5f9da16-e423-4203-a8f7-e275b8b435eb" containerName="payload-processing" Nov 03 17:14:12.852279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852236 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f9da16-e423-4203-a8f7-e275b8b435eb" containerName="payload-processing" Nov 03 17:14:12.852279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852257 6001 cpu_manager.go:414] "containerMap: removing container" podUID="73810920-f5ae-4a5b-bebc-77df1898b031" containerName="revoke-keys" Nov 03 17:14:12.852279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852262 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73810920-f5ae-4a5b-bebc-77df1898b031" containerName="revoke-keys" Nov 03 17:14:12.852279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852274 6001 cpu_manager.go:414] "containerMap: removing container" podUID="25523fec-a8b2-4a74-b9e0-525b3d18e642" containerName="payload-processing" Nov 03 17:14:12.852279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852279 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25523fec-a8b2-4a74-b9e0-525b3d18e642" containerName="payload-processing" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852288 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4fe4815e-29e6-48c8-a035-ef1c00bc571c" containerName="revoke-keys" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852293 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe4815e-29e6-48c8-a035-ef1c00bc571c" containerName="revoke-keys" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852456 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f9da16-e423-4203-a8f7-e275b8b435eb" containerName="payload-processing" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852468 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="73810920-f5ae-4a5b-bebc-77df1898b031" containerName="revoke-keys" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852477 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="25523fec-a8b2-4a74-b9e0-525b3d18e642" containerName="payload-processing" Nov 03 17:14:12.852484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.852483 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe4815e-29e6-48c8-a035-ef1c00bc571c" containerName="revoke-keys" Nov 03 17:14:12.858132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.858114 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:12.874556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.874526 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:14:12.883027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.883003 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:14:12.883271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.883253 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:14:12.883501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.883485 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:14:12.884060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.884036 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk"] Nov 03 17:14:12.988199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.988167 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:12.988461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.988415 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgmdl\" (UniqueName: \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:12.988602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:12.988541 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.090211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.090177 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgmdl\" (UniqueName: \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.090375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.090216 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.090420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.090390 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.091033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.091006 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.092563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.092543 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.114457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.114382 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgmdl\" (UniqueName: \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\") pod \"maas-api-revoke-sub-5e82e77e-5mjlk\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.176889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.176853 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:13.487794 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:13.487765 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda161ffb3_a2bf_44c5_9727_f15d9109b8a4.slice/crio-efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5 WatchSource:0}: Error finding container efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5: Status 404 returned error can't find the container with id efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5 Nov 03 17:14:13.490346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.490317 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk"] Nov 03 17:14:13.844262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.844225 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" event={"ID":"a161ffb3-a2bf-44c5-9727-f15d9109b8a4","Type":"ContainerStarted","Data":"ba6ec013a496554bcdbc4d632a881716d8df45dc2ba20f0fb1f7d059c2e23ed0"} Nov 03 17:14:13.844460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.844293 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" event={"ID":"a161ffb3-a2bf-44c5-9727-f15d9109b8a4","Type":"ContainerStarted","Data":"efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5"} Nov 03 17:14:13.888500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:13.888426 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" podStartSLOduration=1.888413952 podStartE2EDuration="1.888413952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:13.887185016 +0000 UTC m=+3549.220398582" watchObservedRunningTime="2026-11-03 17:14:13.888413952 +0000 UTC m=+3549.221627520" Nov 03 17:14:14.854654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:14.854618 6001 generic.go:356] "Generic (PLEG): container finished" podID="a161ffb3-a2bf-44c5-9727-f15d9109b8a4" containerID="ba6ec013a496554bcdbc4d632a881716d8df45dc2ba20f0fb1f7d059c2e23ed0" exitCode=0 Nov 03 17:14:14.854870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:14.854708 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" event={"ID":"a161ffb3-a2bf-44c5-9727-f15d9109b8a4","Type":"ContainerDied","Data":"ba6ec013a496554bcdbc4d632a881716d8df45dc2ba20f0fb1f7d059c2e23ed0"} Nov 03 17:14:16.076063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.076029 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:16.288466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.261307 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\" (UniqueName: \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\") pod \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " Nov 03 17:14:16.288466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.261364 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\") pod \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " Nov 03 17:14:16.288466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.261419 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\") pod \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\" (UID: \"a161ffb3-a2bf-44c5-9727-f15d9109b8a4\") " Nov 03 17:14:16.288466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.263161 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca" pod "a161ffb3-a2bf-44c5-9727-f15d9109b8a4" (UID: "a161ffb3-a2bf-44c5-9727-f15d9109b8a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:16.312684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.308801 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl" pod "a161ffb3-a2bf-44c5-9727-f15d9109b8a4" (UID: "a161ffb3-a2bf-44c5-9727-f15d9109b8a4"). InnerVolumeSpecName "kube-api-access-xgmdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:16.401469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.365516 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgmdl\" (UniqueName: \"kubernetes.io/projected/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-kube-api-access-xgmdl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:16.401469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.365574 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:16.500984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.500940 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca" pod "a161ffb3-a2bf-44c5-9727-f15d9109b8a4" (UID: "a161ffb3-a2bf-44c5-9727-f15d9109b8a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:16.580610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.580355 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a161ffb3-a2bf-44c5-9727-f15d9109b8a4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:16.877460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.877355 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" Nov 03 17:14:16.877460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.877392 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk" event={"ID":"a161ffb3-a2bf-44c5-9727-f15d9109b8a4","Type":"ContainerDied","Data":"efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5"} Nov 03 17:14:16.877645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:16.877488 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efc2977136491a4e404cd80d610beacae45eae5d1f0e8c371ff87040ef8356e5" Nov 03 17:14:18.032193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:18.032156 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7"] Nov 03 17:14:18.044678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:18.044650 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60079229-phkb7"] Nov 03 17:14:18.792503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:18.792472 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92c4c9c9-98ec-4a59-8bed-7772d187fb77" path="/var/lib/kubelet/pods/92c4c9c9-98ec-4a59-8bed-7772d187fb77/volumes" Nov 03 17:14:19.032989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:19.032951 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw"] Nov 03 17:14:19.046134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:19.046061 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45686da4-4wzlw"] Nov 03 17:14:20.799610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:20.799570 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423bcfdc-07ce-4bcd-882a-d7fc2b668041" path="/var/lib/kubelet/pods/423bcfdc-07ce-4bcd-882a-d7fc2b668041/volumes" Nov 03 17:14:22.254314 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:22.254268 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:14:24.745924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:24.745877 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": context deadline exceeded" Nov 03 17:14:24.834470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:24.817895 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:14:24.946470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:24.931795 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:25.030825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:25.030714 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:14:25.053361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:25.052691 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.751478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.744561 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.777026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.776022 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:14:26.905480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.901521 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.945179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.943732 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:14:26.945179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.944098 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 17:14:26.969201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:26.969152 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:27.059473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.052519 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:27.586573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.586525 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:27.744827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.744784 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:27.745593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.745569 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:14:27.836639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.836528 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:27.837305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.837282 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:14:28.031004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.992814 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:14:28.031004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.993148 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 17:14:28.031004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.993711 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:14:28.031004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:27.993772 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:14:28.061673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:28.053717 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": context deadline exceeded" Nov 03 17:14:28.061673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:28.054399 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:14:28.265509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:28.264994 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:14:28.840475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:28.838947 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:14:28.840475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:28.839924 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Nov 03 17:14:29.047174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.047127 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:14:29.048921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.048881 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded" Nov 03 17:14:29.070860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.028387 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:29.070860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.052596 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 17:14:29.070860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.054287 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:14:29.268660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.268574 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:29.539520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.537768 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:14:29.845089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.844981 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:29.860251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.860191 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:14:29.870140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:29.870105 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:14:30.052472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.051665 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:30.162381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.162289 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:30.242515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.227233 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a161ffb3-a2bf-44c5-9727-f15d9109b8a4" containerName="revoke-keys" Nov 03 17:14:30.242515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.227261 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a161ffb3-a2bf-44c5-9727-f15d9109b8a4" containerName="revoke-keys" Nov 03 17:14:30.242515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.227489 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a161ffb3-a2bf-44c5-9727-f15d9109b8a4" containerName="revoke-keys" Nov 03 17:14:30.279474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.276711 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:30.367739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.367698 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:30.368149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.368126 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:14:30.371937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.368462 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:14:30.371937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.369061 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.397983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.397237 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"openshift-service-ca.crt\"" Nov 03 17:14:30.423313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.423228 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"agc-cred-6922594e2f60\"" Nov 03 17:14:30.424274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.424175 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:14:30.425227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.425071 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:14:30.426757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.426736 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:14:30.439834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.436151 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2492c4\"" Nov 03 17:14:30.439834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.439582 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-e2e--6d96ad03-dockercfg-lhj8z\"" Nov 03 17:14:30.498762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.498570 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"kube-root-ca.crt\"" Nov 03 17:14:30.499330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.499307 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"routing-overlay\"" Nov 03 17:14:30.546468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.545328 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4"] Nov 03 17:14:30.563412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.563381 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba903057-zrtv4"] Nov 03 17:14:30.606496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.603616 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.606496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.603680 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.606496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.603847 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.606496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.603896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfsx\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.734471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.728097 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.744605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.744563 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.759599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.759566 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.771830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.753427 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.771830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.769493 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.772153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.759859 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfsx\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.800099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.799137 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.820134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.818933 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfsx\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:30.829901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.829415 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528d1ac4-2aa8-4301-aa8c-93cc88ff2f35" path="/var/lib/kubelet/pods/528d1ac4-2aa8-4301-aa8c-93cc88ff2f35/volumes" Nov 03 17:14:30.978428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:30.978395 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:31.316048 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:31.316008 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07061875_b062_4b40_8de7_89ff121a1d6c.slice/crio-b5decc7f39973a2d8a504587c96879a3d94bfc4b407dafc4c4cdd6bd443c5a13 WatchSource:0}: Error finding container b5decc7f39973a2d8a504587c96879a3d94bfc4b407dafc4c4cdd6bd443c5a13: Status 404 returned error can't find the container with id b5decc7f39973a2d8a504587c96879a3d94bfc4b407dafc4c4cdd6bd443c5a13 Nov 03 17:14:31.316230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.316212 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:31.392642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.392601 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" event={"ID":"07061875-b062-4b40-8de7-89ff121a1d6c","Type":"ContainerStarted","Data":"b5decc7f39973a2d8a504587c96879a3d94bfc4b407dafc4c4cdd6bd443c5a13"} Nov 03 17:14:31.932486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.932426 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7"] Nov 03 17:14:31.939623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.939584 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:31.959820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.959786 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7"] Nov 03 17:14:31.962995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.962959 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:14:31.962995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.962966 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:14:31.963231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.962967 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:14:31.963231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.963031 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:14:31.997138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.997098 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:31.997581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.997162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:31.997581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:31.997355 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74872\" (UniqueName: \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.098884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.098846 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.099120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.098901 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.099120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.099041 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74872\" (UniqueName: \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.099870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.099836 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.101681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.101641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.128940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.128906 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74872\" (UniqueName: \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\") pod \"maas-api-revoke-sub-daee7470-m6hg7\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.260740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.260697 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:32.553221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.551843 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" event={"ID":"07061875-b062-4b40-8de7-89ff121a1d6c","Type":"ContainerStarted","Data":"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123"} Nov 03 17:14:32.553513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.553493 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:32.614811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.614736 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" podStartSLOduration=3.614714143 podStartE2EDuration="3.614714143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:32.611614126 +0000 UTC m=+3567.944827708" watchObservedRunningTime="2026-11-03 17:14:32.614714143 +0000 UTC m=+3567.947927713" Nov 03 17:14:32.681649 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:32.681609 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-6922594e2f60: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:32.681649 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:32.681641 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:32.681971 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:32.681716 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:33.181685487 +0000 UTC m=+3568.514899051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : secret "agc-cred-6922594e2f60" not found Nov 03 17:14:32.877641 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.877609 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7"] Nov 03 17:14:32.949729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:32.949694 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:33.221471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:33.221361 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-6922594e2f60: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:33.221471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:33.221390 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:33.221471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:33.221475 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:34.221430046 +0000 UTC m=+3569.554643623 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : secret "agc-cred-6922594e2f60" not found Nov 03 17:14:33.606464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:33.601545 6001 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" secret="" err="secret \"payload-processing-external-model-e2e--6d96ad03-dockercfg-lhj8z\" not found" Nov 03 17:14:33.606464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:33.603564 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" event={"ID":"717b27ad-70d0-4b5d-a806-164ef4d28c1a","Type":"ContainerStarted","Data":"0e3c8674f2590df97bfc9f30bd8b4ed4a110c440afa5dd036a69e0754abf1e77"} Nov 03 17:14:33.606464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:33.603623 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" event={"ID":"717b27ad-70d0-4b5d-a806-164ef4d28c1a","Type":"ContainerStarted","Data":"7a07845639ae83e37dada8d7b63f6d6f04687395286c3492583c5d406a1470c7"} Nov 03 17:14:33.676507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:33.667589 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" podStartSLOduration=2.6675694180000002 podStartE2EDuration="2.667569418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:33.666204206 +0000 UTC m=+3568.999417779" watchObservedRunningTime="2026-11-03 17:14:33.667569418 +0000 UTC m=+3569.000782988" Nov 03 17:14:33.756467 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:33.748810 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:33.756467 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:33.748891 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:34.248869355 +0000 UTC m=+3569.582082921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:34.276253 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:34.276219 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-6922594e2f60: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:34.276253 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:34.276251 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:34.276809 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:34.276319 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:36.276298792 +0000 UTC m=+3571.609512362 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : secret "agc-cred-6922594e2f60" not found Nov 03 17:14:34.276809 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:34.276506 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:34.276809 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:34.276590 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:35.276574139 +0000 UTC m=+3570.609787700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:34.699817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:34.699777 6001 generic.go:356] "Generic (PLEG): container finished" podID="717b27ad-70d0-4b5d-a806-164ef4d28c1a" containerID="0e3c8674f2590df97bfc9f30bd8b4ed4a110c440afa5dd036a69e0754abf1e77" exitCode=0 Nov 03 17:14:34.701276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:34.700626 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" event={"ID":"717b27ad-70d0-4b5d-a806-164ef4d28c1a","Type":"ContainerDied","Data":"0e3c8674f2590df97bfc9f30bd8b4ed4a110c440afa5dd036a69e0754abf1e77"} Nov 03 17:14:35.342471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:35.336644 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:35.342471 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:35.336734 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:37.336713213 +0000 UTC m=+3572.669926779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:35.928118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.928091 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.980438 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\") pod \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.980844 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\" (UniqueName: \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\") pod \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.980951 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\") pod \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\" (UID: \"717b27ad-70d0-4b5d-a806-164ef4d28c1a\") " Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.981163 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca" pod "717b27ad-70d0-4b5d-a806-164ef4d28c1a" (UID: "717b27ad-70d0-4b5d-a806-164ef4d28c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.981831 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:35.987535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:35.983913 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872" pod "717b27ad-70d0-4b5d-a806-164ef4d28c1a" (UID: "717b27ad-70d0-4b5d-a806-164ef4d28c1a"). InnerVolumeSpecName "kube-api-access-74872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:36.019540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.004487 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca" pod "717b27ad-70d0-4b5d-a806-164ef4d28c1a" (UID: "717b27ad-70d0-4b5d-a806-164ef4d28c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:36.083549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.083513 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74872\" (UniqueName: \"kubernetes.io/projected/717b27ad-70d0-4b5d-a806-164ef4d28c1a-kube-api-access-74872\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:36.083549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.083541 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717b27ad-70d0-4b5d-a806-164ef4d28c1a-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:36.288895 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:36.288783 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-6922594e2f60: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:36.288895 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:36.288821 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:36.288895 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:36.288891 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:40.288872596 +0000 UTC m=+3575.622086169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : secret "agc-cred-6922594e2f60" not found Nov 03 17:14:36.468834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.468793 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758"] Nov 03 17:14:36.470185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.470163 6001 cpu_manager.go:414] "containerMap: removing container" podUID="717b27ad-70d0-4b5d-a806-164ef4d28c1a" containerName="revoke-keys" Nov 03 17:14:36.470262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.470188 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717b27ad-70d0-4b5d-a806-164ef4d28c1a" containerName="revoke-keys" Nov 03 17:14:36.470472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.470430 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="717b27ad-70d0-4b5d-a806-164ef4d28c1a" containerName="revoke-keys" Nov 03 17:14:36.475847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.475825 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.529497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.525318 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758"] Nov 03 17:14:36.593384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.593294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mjx9\" (UniqueName: \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.593569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.593385 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.593569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.593534 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.695844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.695803 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mjx9\" (UniqueName: \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.696089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.695895 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.696089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.695974 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.697362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.697329 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.699015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.698984 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.720180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.720150 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mjx9\" (UniqueName: \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\") pod \"maas-api-revoke-sub-6eeb0c48-zz758\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:36.727695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.727666 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" Nov 03 17:14:36.727873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.727685 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7" event={"ID":"717b27ad-70d0-4b5d-a806-164ef4d28c1a","Type":"ContainerDied","Data":"7a07845639ae83e37dada8d7b63f6d6f04687395286c3492583c5d406a1470c7"} Nov 03 17:14:36.727873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.727764 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a07845639ae83e37dada8d7b63f6d6f04687395286c3492583c5d406a1470c7" Nov 03 17:14:36.854692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:36.854599 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:37.172701 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:37.172580 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134fcd64_e2c1_4306_8a25_287f9e1572c1.slice/crio-81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8 WatchSource:0}: Error finding container 81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8: Status 404 returned error can't find the container with id 81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8 Nov 03 17:14:37.175635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:37.175608 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758"] Nov 03 17:14:37.411287 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:37.411253 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:37.411487 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:37.411331 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:41.411316722 +0000 UTC m=+3576.744530293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:14:37.739591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:37.739557 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" event={"ID":"134fcd64-e2c1-4306-8a25-287f9e1572c1","Type":"ContainerStarted","Data":"2aa43d3a30e7f21b09008bb6884b0ed100a6568b7916387034341f4519fc722f"} Nov 03 17:14:37.740072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:37.739626 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" event={"ID":"134fcd64-e2c1-4306-8a25-287f9e1572c1","Type":"ContainerStarted","Data":"81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8"} Nov 03 17:14:37.796809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:37.796750 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" podStartSLOduration=1.796732853 podStartE2EDuration="1.796732853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:37.796005593 +0000 UTC m=+3573.129219167" watchObservedRunningTime="2026-11-03 17:14:37.796732853 +0000 UTC m=+3573.129946422" Nov 03 17:14:38.750502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:38.750424 6001 generic.go:356] "Generic (PLEG): container finished" podID="134fcd64-e2c1-4306-8a25-287f9e1572c1" containerID="2aa43d3a30e7f21b09008bb6884b0ed100a6568b7916387034341f4519fc722f" exitCode=0 Nov 03 17:14:38.750956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:38.750508 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" event={"ID":"134fcd64-e2c1-4306-8a25-287f9e1572c1","Type":"ContainerDied","Data":"2aa43d3a30e7f21b09008bb6884b0ed100a6568b7916387034341f4519fc722f"} Nov 03 17:14:39.709469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.702053 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" podUID="07061875-b062-4b40-8de7-89ff121a1d6c" containerName="payload-processing" containerID="cri-o://5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123" gracePeriod=25 Nov 03 17:14:39.808466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.807759 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:14:39.872467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.865698 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:39.952706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.925903 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-24cb6d-dockercfg-sv2kp\"" Nov 03 17:14:39.952706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.925975 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-24cb6d\"" Nov 03 17:14:39.954058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.954028 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:14:39.996257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.990647 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:39.996257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:39.990927 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9djm\" (UniqueName: \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.102127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.095275 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9djm\" (UniqueName: \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.102127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.095620 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.102127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.102040 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.125644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.125582 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9djm\" (UniqueName: \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\") pod \"payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.302123 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:40.302071 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-6922594e2f60: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:40.302224 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:40.302127 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw: secret "agc-cred-6922594e2f60" not found Nov 03 17:14:40.302224 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:40.302197 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials podName:07061875-b062-4b40-8de7-89ff121a1d6c nodeName:}" failed. No retries permitted until 2026-11-03 17:14:48.302176938 +0000 UTC m=+3583.635390524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c") : secret "agc-cred-6922594e2f60" not found Nov 03 17:14:40.335883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.335855 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:40.339747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.339722 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:40.445036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.444226 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446495 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\") pod \"07061875-b062-4b40-8de7-89ff121a1d6c\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446600 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\") pod \"07061875-b062-4b40-8de7-89ff121a1d6c\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446646 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\") pod \"07061875-b062-4b40-8de7-89ff121a1d6c\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446685 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\") pod \"134fcd64-e2c1-4306-8a25-287f9e1572c1\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446717 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\") pod \"07061875-b062-4b40-8de7-89ff121a1d6c\" (UID: \"07061875-b062-4b40-8de7-89ff121a1d6c\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446755 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\" (UniqueName: \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\") pod \"134fcd64-e2c1-4306-8a25-287f9e1572c1\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " Nov 03 17:14:40.448024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.446824 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\") pod \"134fcd64-e2c1-4306-8a25-287f9e1572c1\" (UID: \"134fcd64-e2c1-4306-8a25-287f9e1572c1\") " Nov 03 17:14:40.449069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.448761 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca" pod "134fcd64-e2c1-4306-8a25-287f9e1572c1" (UID: "134fcd64-e2c1-4306-8a25-287f9e1572c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:40.490564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.484231 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume" pod "07061875-b062-4b40-8de7-89ff121a1d6c" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:40.490941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.490893 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume" pod "07061875-b062-4b40-8de7-89ff121a1d6c" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:40.505228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.505184 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca" pod "134fcd64-e2c1-4306-8a25-287f9e1572c1" (UID: "134fcd64-e2c1-4306-8a25-287f9e1572c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:40.505376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.505267 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials" pod "07061875-b062-4b40-8de7-89ff121a1d6c" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:40.505582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.505566 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9" pod "134fcd64-e2c1-4306-8a25-287f9e1572c1" (UID: "134fcd64-e2c1-4306-8a25-287f9e1572c1"). InnerVolumeSpecName "kube-api-access-8mjx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:40.506592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.506565 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx" pod "07061875-b062-4b40-8de7-89ff121a1d6c" (UID: "07061875-b062-4b40-8de7-89ff121a1d6c"). InnerVolumeSpecName "kube-api-access-fdfsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550371 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mjx9\" (UniqueName: \"kubernetes.io/projected/134fcd64-e2c1-4306-8a25-287f9e1572c1-kube-api-access-8mjx9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550406 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550420 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550435 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550465 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfsx\" (UniqueName: \"kubernetes.io/projected/07061875-b062-4b40-8de7-89ff121a1d6c-kube-api-access-fdfsx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550478 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/134fcd64-e2c1-4306-8a25-287f9e1572c1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.550491 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07061875-b062-4b40-8de7-89ff121a1d6c-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.822073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.822038 6001 generic.go:356] "Generic (PLEG): container finished" podID="07061875-b062-4b40-8de7-89ff121a1d6c" containerID="5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123" exitCode=0 Nov 03 17:14:40.822852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.822107 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" Nov 03 17:14:40.822852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.822139 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" event={"ID":"07061875-b062-4b40-8de7-89ff121a1d6c","Type":"ContainerDied","Data":"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123"} Nov 03 17:14:40.822852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.822219 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw" event={"ID":"07061875-b062-4b40-8de7-89ff121a1d6c","Type":"ContainerDied","Data":"b5decc7f39973a2d8a504587c96879a3d94bfc4b407dafc4c4cdd6bd443c5a13"} Nov 03 17:14:40.822852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.822274 6001 scope.go:122] "RemoveContainer" containerID="5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123" Nov 03 17:14:40.826598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.825520 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" event={"ID":"134fcd64-e2c1-4306-8a25-287f9e1572c1","Type":"ContainerDied","Data":"81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8"} Nov 03 17:14:40.826598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.825588 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81ea97c728997540103efb1d208291c50ecd7cbf5c0f57894a5e2883f7a23ec8" Nov 03 17:14:40.826598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.825657 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758" Nov 03 17:14:40.831283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.831258 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:14:40.842724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.841742 6001 scope.go:122] "RemoveContainer" containerID="5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123" Nov 03 17:14:40.845605 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:40.843181 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123\": container with ID starting with 5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123 not found: ID does not exist" containerID="5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123" Nov 03 17:14:40.845605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.843221 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123"} err="failed to get container status \"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123\": rpc error: code = NotFound desc = could not find container \"5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123\": container with ID starting with 5fee0730eb64b9a48e74ea331e333bd904feb56f23da312cb8679caffad8c123 not found: ID does not exist" Nov 03 17:14:40.898740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.898702 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:40.911348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:40.911312 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-6b6fhhjqw"] Nov 03 17:14:41.836982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:41.836940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" event={"ID":"e97485b6-2f22-4247-9c43-992406601b59","Type":"ContainerStarted","Data":"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2"} Nov 03 17:14:41.837407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:41.837038 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" event={"ID":"e97485b6-2f22-4247-9c43-992406601b59","Type":"ContainerStarted","Data":"d7ba64fdaaed51973f41b9cd68d564ee25086928a955200f38b4ffb906ecd695"} Nov 03 17:14:41.837407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:41.837073 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:41.883647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:41.883589 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" podStartSLOduration=2.883573563 podStartE2EDuration="2.883573563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:41.879609248 +0000 UTC m=+3577.212822820" watchObservedRunningTime="2026-11-03 17:14:41.883573563 +0000 UTC m=+3577.216787132" Nov 03 17:14:42.056761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.056729 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7"] Nov 03 17:14:42.063921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.063891 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e65c5ac3-h2gl7"] Nov 03 17:14:42.792423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.792384 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07061875-b062-4b40-8de7-89ff121a1d6c" path="/var/lib/kubelet/pods/07061875-b062-4b40-8de7-89ff121a1d6c/volumes" Nov 03 17:14:42.792920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.792904 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc48d37-2ec7-4036-ac13-e742278ae3b8" path="/var/lib/kubelet/pods/ffc48d37-2ec7-4036-ac13-e742278ae3b8/volumes" Nov 03 17:14:42.871188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.871155 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48"] Nov 03 17:14:42.872040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872024 6001 cpu_manager.go:414] "containerMap: removing container" podUID="134fcd64-e2c1-4306-8a25-287f9e1572c1" containerName="revoke-keys" Nov 03 17:14:42.872040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872041 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134fcd64-e2c1-4306-8a25-287f9e1572c1" containerName="revoke-keys" Nov 03 17:14:42.872136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872077 6001 cpu_manager.go:414] "containerMap: removing container" podUID="07061875-b062-4b40-8de7-89ff121a1d6c" containerName="payload-processing" Nov 03 17:14:42.872136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872083 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07061875-b062-4b40-8de7-89ff121a1d6c" containerName="payload-processing" Nov 03 17:14:42.872217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872208 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="134fcd64-e2c1-4306-8a25-287f9e1572c1" containerName="revoke-keys" Nov 03 17:14:42.872252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.872226 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="07061875-b062-4b40-8de7-89ff121a1d6c" containerName="payload-processing" Nov 03 17:14:42.876504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.876485 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:42.894791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.892950 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:14:42.894791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.892960 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:14:42.894791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.892963 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:14:42.894791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.892964 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:14:42.895065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:42.895046 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48"] Nov 03 17:14:43.010614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.010577 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xck8f\" (UniqueName: \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.010777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.010636 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.010873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.010855 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.113170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.113072 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.113345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.113273 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xck8f\" (UniqueName: \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.113345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.113306 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.114040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.114019 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.115625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.115599 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.138076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.138045 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xck8f\" (UniqueName: \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\") pod \"maas-api-revoke-sub-7f558d86-prd48\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.195356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.195319 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:43.506288 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:43.506253 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70a7aee_b98e_412b_a659_d8d1f4a3fc1b.slice/crio-2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8 WatchSource:0}: Error finding container 2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8: Status 404 returned error can't find the container with id 2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8 Nov 03 17:14:43.509665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.509637 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48"] Nov 03 17:14:43.860076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:43.859989 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" event={"ID":"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b","Type":"ContainerStarted","Data":"2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8"} Nov 03 17:14:44.871612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:44.871577 6001 generic.go:356] "Generic (PLEG): container finished" podID="b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" containerID="883694032f8232f5d9fbb3e1bfa48550ab4c1d8bd9872defb6eeb34d1e983576" exitCode=0 Nov 03 17:14:44.872020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:44.871655 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" event={"ID":"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b","Type":"ContainerDied","Data":"883694032f8232f5d9fbb3e1bfa48550ab4c1d8bd9872defb6eeb34d1e983576"} Nov 03 17:14:46.133644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.133613 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:46.321361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.321322 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\") pod \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " Nov 03 17:14:46.321634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.321549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\" (UniqueName: \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\") pod \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " Nov 03 17:14:46.321757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.321731 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca" pod "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" (UID: "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:46.321828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.321744 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\") pod \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\" (UID: \"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b\") " Nov 03 17:14:46.322438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.322420 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:46.323775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.323746 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca" pod "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" (UID: "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:14:46.323874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.323855 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f" pod "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" (UID: "b70a7aee-b98e-412b-a659-d8d1f4a3fc1b"). InnerVolumeSpecName "kube-api-access-xck8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:46.423986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.423947 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:46.423986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.423985 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xck8f\" (UniqueName: \"kubernetes.io/projected/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b-kube-api-access-xck8f\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:14:46.712298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.712208 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:14:46.713290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.713265 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" containerName="revoke-keys" Nov 03 17:14:46.713290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.713288 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" containerName="revoke-keys" Nov 03 17:14:46.713621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.713603 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" containerName="revoke-keys" Nov 03 17:14:46.721488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.717895 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:46.727948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.727918 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:46.728161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.728144 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56v6j\" (UniqueName: \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:46.734910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.734887 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-44736d-dockercfg-7c7hj\"" Nov 03 17:14:46.735048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.734937 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-44736d\"" Nov 03 17:14:46.736425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.736390 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:14:46.752241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.752203 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:14:46.759262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.759232 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:46.791970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:46.777819 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:14:47.073397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.073357 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.093190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.093151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:47.093684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.093654 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9c6\" (UniqueName: \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.094081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.094059 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56v6j\" (UniqueName: \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:47.095665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.095641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:47.155482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.155431 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56v6j\" (UniqueName: \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\") pod \"payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:47.156046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.156027 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" Nov 03 17:14:47.158064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.158033 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48" event={"ID":"b70a7aee-b98e-412b-a659-d8d1f4a3fc1b","Type":"ContainerDied","Data":"2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8"} Nov 03 17:14:47.158191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.158089 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c569776bdf091dd91a5336d58d7e9cdece97af9330e5dc1f072164b7abb89c8" Nov 03 17:14:47.201103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.201064 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.201312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.201267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9c6\" (UniqueName: \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.202345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.202322 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.225972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.225945 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9c6\" (UniqueName: \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\") pod \"payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.330366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.330285 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:47.338757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.336610 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:47.647256 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:47.647216 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab40dc7_5ae9_4a08_9c01_f15227c21f82.slice/crio-6b4a17439b1724c8f72fa44ec6728ae5e3651c6f41cc7262f5ea215948e864e1 WatchSource:0}: Error finding container 6b4a17439b1724c8f72fa44ec6728ae5e3651c6f41cc7262f5ea215948e864e1: Status 404 returned error can't find the container with id 6b4a17439b1724c8f72fa44ec6728ae5e3651c6f41cc7262f5ea215948e864e1 Nov 03 17:14:47.649078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.649000 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:14:47.661561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:47.661508 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:14:47.668055 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:14:47.668031 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f71b6a_a26b_4648_8388_7e28f656537a.slice/crio-1c5f223f9a011570ab8d017b23ca68bef214f5ecbd1afc576ebc7cb8153221bd WatchSource:0}: Error finding container 1c5f223f9a011570ab8d017b23ca68bef214f5ecbd1afc576ebc7cb8153221bd: Status 404 returned error can't find the container with id 1c5f223f9a011570ab8d017b23ca68bef214f5ecbd1afc576ebc7cb8153221bd Nov 03 17:14:48.179471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.178410 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" event={"ID":"22f71b6a-a26b-4648-8388-7e28f656537a","Type":"ContainerStarted","Data":"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f"} Nov 03 17:14:48.179471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.178514 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" event={"ID":"22f71b6a-a26b-4648-8388-7e28f656537a","Type":"ContainerStarted","Data":"1c5f223f9a011570ab8d017b23ca68bef214f5ecbd1afc576ebc7cb8153221bd"} Nov 03 17:14:48.179471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.178566 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:14:48.182155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.182128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" event={"ID":"fab40dc7-5ae9-4a08-9c01-f15227c21f82","Type":"ContainerStarted","Data":"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5"} Nov 03 17:14:48.183290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.183277 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:14:48.183437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.183423 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" event={"ID":"fab40dc7-5ae9-4a08-9c01-f15227c21f82","Type":"ContainerStarted","Data":"6b4a17439b1724c8f72fa44ec6728ae5e3651c6f41cc7262f5ea215948e864e1"} Nov 03 17:14:48.234769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.234713 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" podStartSLOduration=2.234696469 podStartE2EDuration="2.234696469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:48.228621472 +0000 UTC m=+3583.561835042" watchObservedRunningTime="2026-11-03 17:14:48.234696469 +0000 UTC m=+3583.567910048" Nov 03 17:14:48.279328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:48.279273 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" podStartSLOduration=2.279256794 podStartE2EDuration="2.279256794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:48.275632062 +0000 UTC m=+3583.608845632" watchObservedRunningTime="2026-11-03 17:14:48.279256794 +0000 UTC m=+3583.612470362" Nov 03 17:14:52.254820 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:14:52.254782 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:14:52.852412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:52.852382 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:14:59.196265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:14:59.196235 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:15:00.182285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.182250 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:00.188067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.188042 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.198341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.198306 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:00.207318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.207292 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:15:00.207540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.207340 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:00.207540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.207370 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.211955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.210252 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:15:00.214701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.214670 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:00.214926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.214900 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.221068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.221044 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:00.221211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.221173 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.243162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.243124 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:15:00.243381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.243349 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:15:00.298695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.298658 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:00.298912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.298713 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:00.298912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.298739 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:00.298912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.298833 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.300176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.299702 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:00.309873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.309836 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:00.315629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.315607 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.318051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.318031 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:00.323748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.323571 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.330783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.330755 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340417 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwf6\" (UniqueName: \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340516 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgb9t\" (UniqueName: \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340644 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktnp6\" (UniqueName: \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340718 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340812 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340868 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfw5\" (UniqueName: \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.340950 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.341966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.341026 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.348061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.348036 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.353684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.353663 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:00.361854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.361672 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.364425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.362620 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:00.369665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.369377 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.381553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.381521 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:00.395281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.395252 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:00.401395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.401375 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:00.406864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.406845 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:00.418654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.418633 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:00.447210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.447123 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.447491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.447463 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.447663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.447646 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.456465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.447863 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwf6\" (UniqueName: \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.461467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.451176 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.461467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.451817 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.461467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.456803 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:00.469649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.469618 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxszr\" (UniqueName: \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.469994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.469969 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxczc\" (UniqueName: \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.470136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.470119 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.470297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.470280 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgb9t\" (UniqueName: \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.470412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.470397 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.481029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.480992 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-446nj\" (UniqueName: \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.481837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.481316 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:00.481837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.481381 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.482639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.482620 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.482860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.482843 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktnp6\" (UniqueName: \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.483297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.483278 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.484787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.484750 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.484975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.484959 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c4jw\" (UniqueName: \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.485077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.485065 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.485207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.485196 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.485293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.485284 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzq97\" (UniqueName: \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.485362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.485354 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfw5\" (UniqueName: \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.485455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.485430 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.486264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.486244 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.487054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.487034 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.488850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.488828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwf6\" (UniqueName: \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.490591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.490571 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:00.490969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.490950 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.497106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.497083 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:00.497203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.497188 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.500941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.500878 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:00.503357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.503331 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgb9t\" (UniqueName: \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.508989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.507972 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:00.509161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.509141 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:00.509921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.509897 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktnp6\" (UniqueName: \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.537939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.537908 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:00.545170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.545136 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfw5\" (UniqueName: \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\") pod \"maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.550186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.550116 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:00.552195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.552168 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:00.559575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.559430 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.561182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.561151 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:00.566520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.566432 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:00.568435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.568414 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:00.568627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.568610 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.577172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.577141 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:00.587277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587249 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxszr\" (UniqueName: \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.587395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587299 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxczc\" (UniqueName: \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.587395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587330 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.587395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587378 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.587631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587407 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-446nj\" (UniqueName: \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.587631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587438 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxcs\" (UniqueName: \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.587631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587489 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.587631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587582 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587631 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6t8p\" (UniqueName: \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587667 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587708 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24sk9\" (UniqueName: \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587760 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587798 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86tm\" (UniqueName: \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587839 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2tr4\" (UniqueName: \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587891 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c4jw\" (UniqueName: \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587918 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.587998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.587983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.588464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.588056 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzq97\" (UniqueName: \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.588464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.588116 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.588464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.588236 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.588464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.588322 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.588683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.588466 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.589612 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.589648 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.590213 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.590765 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.590875 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.591232 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.593852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.593530 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:00.598767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.598739 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:00.617497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.606713 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:00.617497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.610029 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.620844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.620705 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.622912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.622888 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:00.631787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.631278 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxszr\" (UniqueName: \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.632598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.632553 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxczc\" (UniqueName: \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.632598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.632567 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.635756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.633378 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c4jw\" (UniqueName: \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.635756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.635420 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzq97\" (UniqueName: \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.636155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.636119 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-446nj\" (UniqueName: \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.638009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.637842 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:00.645351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.645210 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:00.645351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.645283 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.652532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.652474 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:00.660588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.660525 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:00.666296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.666083 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:00.695914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.695314 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.695914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.695367 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.695914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.695412 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6bf\" (UniqueName: \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.696788 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6t8p\" (UniqueName: \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.696851 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.696894 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24sk9\" (UniqueName: \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.696948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86tm\" (UniqueName: \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.696993 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2tr4\" (UniqueName: \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697078 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697311 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpmh6\" (UniqueName: \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697356 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.697525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697500 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxcs\" (UniqueName: \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.698768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.697539 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.698826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.698805 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.704195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.699073 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.704195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.699645 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.704195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.700961 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.704195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.702070 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.704195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.699990 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:00.710249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.710227 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:00.729417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.728940 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxcs\" (UniqueName: \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.729417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.729142 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:00.731346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.730799 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86tm\" (UniqueName: \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:00.732368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.731921 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24sk9\" (UniqueName: \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.739178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.739152 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2tr4\" (UniqueName: \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.739726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.739243 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6t8p\" (UniqueName: \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.761858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.761569 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:00.769719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.769542 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:00.803896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.803826 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.804708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.804667 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpmh6\" (UniqueName: \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.807008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.806989 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6tp\" (UniqueName: \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.807224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.807211 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.807311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.807300 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6bf\" (UniqueName: \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.807390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.807381 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.809039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.808915 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.809039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.808982 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.823501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.820396 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:00.846814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.846667 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpmh6\" (UniqueName: \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\") pod \"maas-api-key-cleanup-29895435-m6l29\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.879288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.879217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6bf\" (UniqueName: \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.910518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.908476 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.910518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.908677 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6tp\" (UniqueName: \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.910518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.909991 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:00.914908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.914742 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:00.918248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.918222 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:00.920698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.920645 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:00.931925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.931753 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:00.946249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:00.946176 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6tp\" (UniqueName: \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:01.037334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.037298 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:01.082852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.082746 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:01.136352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.136051 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:01.379221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.379137 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:01.414369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.414313 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:01.487530 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.487428 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b7b2e1_019e_4b2c_bfa5_4ef2397718c7.slice/crio-d5451045d565b6e384e79c12967a8588b58db85b7f983fee6b57ffc2135243ca WatchSource:0}: Error finding container d5451045d565b6e384e79c12967a8588b58db85b7f983fee6b57ffc2135243ca: Status 404 returned error can't find the container with id d5451045d565b6e384e79c12967a8588b58db85b7f983fee6b57ffc2135243ca Nov 03 17:15:01.564905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.564866 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerStarted","Data":"d5451045d565b6e384e79c12967a8588b58db85b7f983fee6b57ffc2135243ca"} Nov 03 17:15:01.566414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.566370 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerStarted","Data":"79dd8a201263f881b879c2c4b5ea10bc7754d5bd5b359adef1034bb2b2bc6dba"} Nov 03 17:15:01.567950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.567921 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerStarted","Data":"7225111f2bee4d1b4f6844437fa5b1b531a21e83f5c197c8294a94ebe5d3be1d"} Nov 03 17:15:01.592605 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.592575 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c64aac_55d0_4f28_8827_fa11441e5481.slice/crio-db423c8cea2304d52764ded7c47b1b9f27c5a99bcb9f3667a870527f70257872 WatchSource:0}: Error finding container db423c8cea2304d52764ded7c47b1b9f27c5a99bcb9f3667a870527f70257872: Status 404 returned error can't find the container with id db423c8cea2304d52764ded7c47b1b9f27c5a99bcb9f3667a870527f70257872 Nov 03 17:15:01.597516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.597483 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:01.629807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.629662 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:01.639194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.639164 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:01.699674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.699637 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:01.706301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.706257 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:01.713275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.713230 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:01.719027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.718951 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:01.753363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.753270 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:01.759413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.759388 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:01.765647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.765625 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:01.866530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.866505 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:01.882307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.882282 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:01.887015 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.886986 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a601ed0_5b71_42a3_8974_5d7c0770ffb1.slice/crio-741aed863538ac0c6458bee2b96113c343ef927135281587a9b4202055cd7d71 WatchSource:0}: Error finding container 741aed863538ac0c6458bee2b96113c343ef927135281587a9b4202055cd7d71: Status 404 returned error can't find the container with id 741aed863538ac0c6458bee2b96113c343ef927135281587a9b4202055cd7d71 Nov 03 17:15:01.888334 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.888202 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90629a0b_74c3_477e_9468_72681dd5b78d.slice/crio-b71f47e76518338bff6e37549b44e7321e11153a4f4a3a640dee5ce5e699004b WatchSource:0}: Error finding container b71f47e76518338bff6e37549b44e7321e11153a4f4a3a640dee5ce5e699004b: Status 404 returned error can't find the container with id b71f47e76518338bff6e37549b44e7321e11153a4f4a3a640dee5ce5e699004b Nov 03 17:15:01.888718 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.888673 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91179a4_e019_4998_b071_736a3c0f4ab7.slice/crio-563867ec7f35c61f343a4d9ab60b88d61acc0c77d2cad3c1b890bb6153d87671 WatchSource:0}: Error finding container 563867ec7f35c61f343a4d9ab60b88d61acc0c77d2cad3c1b890bb6153d87671: Status 404 returned error can't find the container with id 563867ec7f35c61f343a4d9ab60b88d61acc0c77d2cad3c1b890bb6153d87671 Nov 03 17:15:01.891436 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.891347 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee7480b_8265_4acb_b9e3_b61dfb2b6cef.slice/crio-596ea314d296c1556e2875aef3134e6d52f53339c8256802383699bbf1122503 WatchSource:0}: Error finding container 596ea314d296c1556e2875aef3134e6d52f53339c8256802383699bbf1122503: Status 404 returned error can't find the container with id 596ea314d296c1556e2875aef3134e6d52f53339c8256802383699bbf1122503 Nov 03 17:15:01.891436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.891409 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:01.894693 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.894668 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad571d0_2095_4b3a_9c41_bd4b393fdbfd.slice/crio-a8bf861b9cb98d00456f0bcd7b8118facfb2e69fc4e6bd05ede8e42db2636682 WatchSource:0}: Error finding container a8bf861b9cb98d00456f0bcd7b8118facfb2e69fc4e6bd05ede8e42db2636682: Status 404 returned error can't find the container with id a8bf861b9cb98d00456f0bcd7b8118facfb2e69fc4e6bd05ede8e42db2636682 Nov 03 17:15:01.898310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.898274 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:01.906643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:01.906587 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:01.918980 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:01.918958 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb33eb94_227c_4d5c_bf79_c6692cd369a1.slice/crio-b1475863667d1af60521c987a5c51eba65ba54e9756f39d6421842dbecd87739 WatchSource:0}: Error finding container b1475863667d1af60521c987a5c51eba65ba54e9756f39d6421842dbecd87739: Status 404 returned error can't find the container with id b1475863667d1af60521c987a5c51eba65ba54e9756f39d6421842dbecd87739 Nov 03 17:15:02.579522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.579482 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerStarted","Data":"dfa5174c49e29ee2248220c6a0aa35ac583ff8466556cfe523e766d531564d00"} Nov 03 17:15:02.580938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.580915 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerStarted","Data":"7db4394639362967f2b7daece03d79c61441ca6ac5747a7b989eaacd5460a720"} Nov 03 17:15:02.582221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.582198 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerStarted","Data":"47121ab85371340ea45f9bea5d3087a6c545c233a9b1abb5da1638f7ae2acd3e"} Nov 03 17:15:02.583550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.583526 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerStarted","Data":"b1475863667d1af60521c987a5c51eba65ba54e9756f39d6421842dbecd87739"} Nov 03 17:15:02.584803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.584778 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerStarted","Data":"0e5bb5ac1210b54168538482db19b5a5728117225ec82117f1df0b2ea158197c"} Nov 03 17:15:02.586032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.586007 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerStarted","Data":"596ea314d296c1556e2875aef3134e6d52f53339c8256802383699bbf1122503"} Nov 03 17:15:02.587257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.587230 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerStarted","Data":"9f4e25178d54f969b0bf24cacd0e578df6367b51826f8356f15f34554bf60f40"} Nov 03 17:15:02.588572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.588550 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerStarted","Data":"a8bf861b9cb98d00456f0bcd7b8118facfb2e69fc4e6bd05ede8e42db2636682"} Nov 03 17:15:02.589776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.589754 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerStarted","Data":"741aed863538ac0c6458bee2b96113c343ef927135281587a9b4202055cd7d71"} Nov 03 17:15:02.591004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.590984 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerStarted","Data":"cdcbc726abb35fc598527f56ed37aa496386a2627dec5f70ce4497ff545ab913"} Nov 03 17:15:02.592490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.592469 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerStarted","Data":"00f4b2989d3a1afab49195ee12a2e3ddeac809b7bf63a680cb5bfee6b26f640b"} Nov 03 17:15:02.594082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.594059 6001 generic.go:356] "Generic (PLEG): container finished" podID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerID="285f2c7a435e5d48e83671333371bca1152aa23c31995d8a66fc18e7fa9268c4" exitCode=60 Nov 03 17:15:02.594167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.594122 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerDied","Data":"285f2c7a435e5d48e83671333371bca1152aa23c31995d8a66fc18e7fa9268c4"} Nov 03 17:15:02.594492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.594476 6001 scope.go:122] "RemoveContainer" containerID="285f2c7a435e5d48e83671333371bca1152aa23c31995d8a66fc18e7fa9268c4" Nov 03 17:15:02.595497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.595434 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerStarted","Data":"b71f47e76518338bff6e37549b44e7321e11153a4f4a3a640dee5ce5e699004b"} Nov 03 17:15:02.596826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.596787 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerStarted","Data":"db423c8cea2304d52764ded7c47b1b9f27c5a99bcb9f3667a870527f70257872"} Nov 03 17:15:02.597982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.597963 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerStarted","Data":"dd46831d68c81beaceb474c397da1f6fef50157c39ed2288e16c05cc146b3bed"} Nov 03 17:15:02.599370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.599348 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerStarted","Data":"563867ec7f35c61f343a4d9ab60b88d61acc0c77d2cad3c1b890bb6153d87671"} Nov 03 17:15:02.600643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.600624 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerStarted","Data":"809510a94e7c0bf0e46e908e1c6c8a8036b035f0eaab60a8432afb5d465acb05"} Nov 03 17:15:02.637307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:02.637198 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" podStartSLOduration=2.637183042 podStartE2EDuration="2.637183042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:02.635871312 +0000 UTC m=+3597.969084881" watchObservedRunningTime="2026-11-03 17:15:02.637183042 +0000 UTC m=+3597.970396651" Nov 03 17:15:03.612653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.612542 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerStarted","Data":"21c93e7872d93355ec6328cdcba1c35b1702fb21293b13d2b9413517250f288d"} Nov 03 17:15:03.614516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.614477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerStarted","Data":"204327f1ee56745cc5170c781681081759ab87aa741e7bf746d205716ad1ff67"} Nov 03 17:15:03.616252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.616220 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerStarted","Data":"4c1f8c8af8a3ed5993d9b510c952b6d4e2fd607c311a3b7deb3e4594d5346d7e"} Nov 03 17:15:03.618001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.617972 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerStarted","Data":"e5721ecedcf6bee6796651d3c29415d3f11b3a3da9d6ef3fbbb09a8b239eeb62"} Nov 03 17:15:03.619778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.619745 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerStarted","Data":"d6547bdcab49ef95cd33bd44c0d8ea231e94466a73df1a38d76d297664970f8b"} Nov 03 17:15:03.621617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.621585 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerStarted","Data":"97224ae66339ad4ea5ee1500765b211e8cb2ba2a24c3c57f7ca543b3bdc99f2e"} Nov 03 17:15:03.623350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.623321 6001 generic.go:356] "Generic (PLEG): container finished" podID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerID="f0a4bb4a48608128811eb97c527449b44a8a5a4ff486d76a8ba7638e2aefe577" exitCode=60 Nov 03 17:15:03.623527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.623401 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerDied","Data":"f0a4bb4a48608128811eb97c527449b44a8a5a4ff486d76a8ba7638e2aefe577"} Nov 03 17:15:03.623676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.623657 6001 scope.go:122] "RemoveContainer" containerID="f0a4bb4a48608128811eb97c527449b44a8a5a4ff486d76a8ba7638e2aefe577" Nov 03 17:15:03.625484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.625426 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerID="00f4b2989d3a1afab49195ee12a2e3ddeac809b7bf63a680cb5bfee6b26f640b" exitCode=60 Nov 03 17:15:03.625574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.625505 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerDied","Data":"00f4b2989d3a1afab49195ee12a2e3ddeac809b7bf63a680cb5bfee6b26f640b"} Nov 03 17:15:03.625771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.625756 6001 scope.go:122] "RemoveContainer" containerID="00f4b2989d3a1afab49195ee12a2e3ddeac809b7bf63a680cb5bfee6b26f640b" Nov 03 17:15:03.627817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.627788 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerStarted","Data":"6414734145bf2dc39d92970840d81061159bbd13781db8b1e20c28fd97355c51"} Nov 03 17:15:03.629790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.629679 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerStarted","Data":"307bd76d903dea5f6d8f13a211252d5cd691e53a582d1aeff61cd6469fbcca25"} Nov 03 17:15:03.631488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.631461 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerStarted","Data":"6d555ba5f2fef00616952175a5517892307542241ebf332580159a07d6003435"} Nov 03 17:15:03.661872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.661820 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" podStartSLOduration=3.661803147 podStartE2EDuration="3.661803147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.654835019 +0000 UTC m=+3598.988048609" watchObservedRunningTime="2026-11-03 17:15:03.661803147 +0000 UTC m=+3598.995016763" Nov 03 17:15:03.706075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.706020 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" podStartSLOduration=3.706007068 podStartE2EDuration="3.706007068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.699764812 +0000 UTC m=+3599.032978382" watchObservedRunningTime="2026-11-03 17:15:03.706007068 +0000 UTC m=+3599.039220670" Nov 03 17:15:03.747322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.747248 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" podStartSLOduration=3.747228437 podStartE2EDuration="3.747228437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.742572251 +0000 UTC m=+3599.075785819" watchObservedRunningTime="2026-11-03 17:15:03.747228437 +0000 UTC m=+3599.080442006" Nov 03 17:15:03.791953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.791894 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" podStartSLOduration=3.791878925 podStartE2EDuration="3.791878925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.785704251 +0000 UTC m=+3599.118917819" watchObservedRunningTime="2026-11-03 17:15:03.791878925 +0000 UTC m=+3599.125092566" Nov 03 17:15:03.835597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.835432 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" podStartSLOduration=3.835409266 podStartE2EDuration="3.835409266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.829006197 +0000 UTC m=+3599.162219782" watchObservedRunningTime="2026-11-03 17:15:03.835409266 +0000 UTC m=+3599.168622836" Nov 03 17:15:03.879475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.879367 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" podStartSLOduration=3.879352005 podStartE2EDuration="3.879352005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.87430253 +0000 UTC m=+3599.207516098" watchObservedRunningTime="2026-11-03 17:15:03.879352005 +0000 UTC m=+3599.212565587" Nov 03 17:15:03.920328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:03.920264 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" podStartSLOduration=3.92024834 podStartE2EDuration="3.92024834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.918189184 +0000 UTC m=+3599.251402754" watchObservedRunningTime="2026-11-03 17:15:03.92024834 +0000 UTC m=+3599.253461911" Nov 03 17:15:04.007412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.007350 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" podStartSLOduration=4.007336089 podStartE2EDuration="4.007336089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:04.003165193 +0000 UTC m=+3599.336378763" watchObservedRunningTime="2026-11-03 17:15:04.007336089 +0000 UTC m=+3599.340549657" Nov 03 17:15:04.049186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.049115 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" podStartSLOduration=4.049093641 podStartE2EDuration="4.049093641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:04.046298895 +0000 UTC m=+3599.379512475" watchObservedRunningTime="2026-11-03 17:15:04.049093641 +0000 UTC m=+3599.382307212" Nov 03 17:15:04.831278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.831239 6001 generic.go:356] "Generic (PLEG): container finished" podID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerID="6d555ba5f2fef00616952175a5517892307542241ebf332580159a07d6003435" exitCode=60 Nov 03 17:15:04.834992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.834963 6001 scope.go:122] "RemoveContainer" containerID="6d555ba5f2fef00616952175a5517892307542241ebf332580159a07d6003435" Nov 03 17:15:04.839620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.839079 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerDied","Data":"6d555ba5f2fef00616952175a5517892307542241ebf332580159a07d6003435"} Nov 03 17:15:04.839620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.839144 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerStarted","Data":"e3e0b0dd6f63d4024747b8ef15487ef95e094df98225641464a0fb70ec08f22f"} Nov 03 17:15:04.840169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.840031 6001 scope.go:122] "RemoveContainer" containerID="e3e0b0dd6f63d4024747b8ef15487ef95e094df98225641464a0fb70ec08f22f" Nov 03 17:15:04.843560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.843424 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerStarted","Data":"741cc1b2a22390270b35ab420ee6d33afcb4584468dc398ceea03301b20b750c"} Nov 03 17:15:04.846198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.845995 6001 generic.go:356] "Generic (PLEG): container finished" podID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerID="21c93e7872d93355ec6328cdcba1c35b1702fb21293b13d2b9413517250f288d" exitCode=60 Nov 03 17:15:04.846198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.846086 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerDied","Data":"21c93e7872d93355ec6328cdcba1c35b1702fb21293b13d2b9413517250f288d"} Nov 03 17:15:04.846337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.846325 6001 scope.go:122] "RemoveContainer" containerID="21c93e7872d93355ec6328cdcba1c35b1702fb21293b13d2b9413517250f288d" Nov 03 17:15:04.848933 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.848907 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerID="204327f1ee56745cc5170c781681081759ab87aa741e7bf746d205716ad1ff67" exitCode=60 Nov 03 17:15:04.849057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.848940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerDied","Data":"204327f1ee56745cc5170c781681081759ab87aa741e7bf746d205716ad1ff67"} Nov 03 17:15:04.849272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.849253 6001 scope.go:122] "RemoveContainer" containerID="204327f1ee56745cc5170c781681081759ab87aa741e7bf746d205716ad1ff67" Nov 03 17:15:04.851209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.851189 6001 generic.go:356] "Generic (PLEG): container finished" podID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerID="49571fc4becc4e11b2ba8274fc5e58fed01abe892f7683bb9b90de1291f6aa3e" exitCode=60 Nov 03 17:15:04.851404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.851227 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerDied","Data":"49571fc4becc4e11b2ba8274fc5e58fed01abe892f7683bb9b90de1291f6aa3e"} Nov 03 17:15:04.852003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.851975 6001 scope.go:122] "RemoveContainer" containerID="49571fc4becc4e11b2ba8274fc5e58fed01abe892f7683bb9b90de1291f6aa3e" Nov 03 17:15:04.855171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.855146 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerStarted","Data":"3e025eb18001b5094ae5e825f2e53239dfe5be9438bae7042f9c0d1ca2c692df"} Nov 03 17:15:04.855407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.855389 6001 scope.go:122] "RemoveContainer" containerID="3e025eb18001b5094ae5e825f2e53239dfe5be9438bae7042f9c0d1ca2c692df" Nov 03 17:15:04.857130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.857108 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fcb5379-f832-449f-827e-acce7c4c3285" containerID="4c1f8c8af8a3ed5993d9b510c952b6d4e2fd607c311a3b7deb3e4594d5346d7e" exitCode=60 Nov 03 17:15:04.857210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.857190 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerDied","Data":"4c1f8c8af8a3ed5993d9b510c952b6d4e2fd607c311a3b7deb3e4594d5346d7e"} Nov 03 17:15:04.857410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.857393 6001 scope.go:122] "RemoveContainer" containerID="4c1f8c8af8a3ed5993d9b510c952b6d4e2fd607c311a3b7deb3e4594d5346d7e" Nov 03 17:15:04.859073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.859051 6001 generic.go:356] "Generic (PLEG): container finished" podID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerID="e5721ecedcf6bee6796651d3c29415d3f11b3a3da9d6ef3fbbb09a8b239eeb62" exitCode=60 Nov 03 17:15:04.859181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.859130 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerDied","Data":"e5721ecedcf6bee6796651d3c29415d3f11b3a3da9d6ef3fbbb09a8b239eeb62"} Nov 03 17:15:04.859334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.859320 6001 scope.go:122] "RemoveContainer" containerID="e5721ecedcf6bee6796651d3c29415d3f11b3a3da9d6ef3fbbb09a8b239eeb62" Nov 03 17:15:04.861040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.861022 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerID="d6547bdcab49ef95cd33bd44c0d8ea231e94466a73df1a38d76d297664970f8b" exitCode=60 Nov 03 17:15:04.861115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.861100 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerDied","Data":"d6547bdcab49ef95cd33bd44c0d8ea231e94466a73df1a38d76d297664970f8b"} Nov 03 17:15:04.861299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.861283 6001 scope.go:122] "RemoveContainer" containerID="d6547bdcab49ef95cd33bd44c0d8ea231e94466a73df1a38d76d297664970f8b" Nov 03 17:15:04.863095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.863051 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerID="97224ae66339ad4ea5ee1500765b211e8cb2ba2a24c3c57f7ca543b3bdc99f2e" exitCode=60 Nov 03 17:15:04.863205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.863096 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerDied","Data":"97224ae66339ad4ea5ee1500765b211e8cb2ba2a24c3c57f7ca543b3bdc99f2e"} Nov 03 17:15:04.863375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.863358 6001 scope.go:122] "RemoveContainer" containerID="97224ae66339ad4ea5ee1500765b211e8cb2ba2a24c3c57f7ca543b3bdc99f2e" Nov 03 17:15:04.865032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.865012 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerStarted","Data":"147c3ce3f6caf99eb65a3e911bebe44094f1365f819fad4739c35d1de060d663"} Nov 03 17:15:04.865259 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.865243 6001 scope.go:122] "RemoveContainer" containerID="147c3ce3f6caf99eb65a3e911bebe44094f1365f819fad4739c35d1de060d663" Nov 03 17:15:04.867392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.867369 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerStarted","Data":"b5dde2b8ee63723925f7ea73a4cceca5cca60245942b6a2d63273b161d7185ed"} Nov 03 17:15:04.869567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.869537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerStarted","Data":"cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51"} Nov 03 17:15:04.871175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.871153 6001 generic.go:356] "Generic (PLEG): container finished" podID="90629a0b-74c3-477e-9468-72681dd5b78d" containerID="6414734145bf2dc39d92970840d81061159bbd13781db8b1e20c28fd97355c51" exitCode=60 Nov 03 17:15:04.871269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.871228 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerDied","Data":"6414734145bf2dc39d92970840d81061159bbd13781db8b1e20c28fd97355c51"} Nov 03 17:15:04.871428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.871412 6001 scope.go:122] "RemoveContainer" containerID="6414734145bf2dc39d92970840d81061159bbd13781db8b1e20c28fd97355c51" Nov 03 17:15:04.872947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.872929 6001 generic.go:356] "Generic (PLEG): container finished" podID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerID="307bd76d903dea5f6d8f13a211252d5cd691e53a582d1aeff61cd6469fbcca25" exitCode=60 Nov 03 17:15:04.873013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.872987 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerDied","Data":"307bd76d903dea5f6d8f13a211252d5cd691e53a582d1aeff61cd6469fbcca25"} Nov 03 17:15:04.873178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:04.873165 6001 scope.go:122] "RemoveContainer" containerID="307bd76d903dea5f6d8f13a211252d5cd691e53a582d1aeff61cd6469fbcca25" Nov 03 17:15:05.145958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.145388 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2"] Nov 03 17:15:05.153561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.153526 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d2eebae-6zjz2"] Nov 03 17:15:05.177739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.177589 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" podStartSLOduration=5.177569961 podStartE2EDuration="5.177569961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:05.170982142 +0000 UTC m=+3600.504195712" watchObservedRunningTime="2026-11-03 17:15:05.177569961 +0000 UTC m=+3600.510783532" Nov 03 17:15:05.305056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.304251 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" podStartSLOduration=5.304236959 podStartE2EDuration="5.304236959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:05.297771469 +0000 UTC m=+3600.630985051" watchObservedRunningTime="2026-11-03 17:15:05.304236959 +0000 UTC m=+3600.637450544" Nov 03 17:15:05.414196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.414133 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" podStartSLOduration=5.414116053 podStartE2EDuration="5.414116053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:05.408215842 +0000 UTC m=+3600.741429411" watchObservedRunningTime="2026-11-03 17:15:05.414116053 +0000 UTC m=+3600.747329643" Nov 03 17:15:05.542802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.542725 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:15:05.663165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.663139 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:15:05.886744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.886713 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerID="147c3ce3f6caf99eb65a3e911bebe44094f1365f819fad4739c35d1de060d663" exitCode=60 Nov 03 17:15:05.887146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.886791 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerDied","Data":"147c3ce3f6caf99eb65a3e911bebe44094f1365f819fad4739c35d1de060d663"} Nov 03 17:15:05.888666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.888640 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbe06000-58db-4eeb-86ff-1e604357086c" containerID="b5dde2b8ee63723925f7ea73a4cceca5cca60245942b6a2d63273b161d7185ed" exitCode=60 Nov 03 17:15:05.888809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.888720 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerDied","Data":"b5dde2b8ee63723925f7ea73a4cceca5cca60245942b6a2d63273b161d7185ed"} Nov 03 17:15:05.889043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.889019 6001 scope.go:122] "RemoveContainer" containerID="b5dde2b8ee63723925f7ea73a4cceca5cca60245942b6a2d63273b161d7185ed" Nov 03 17:15:05.891050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.891029 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerStarted","Data":"fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687"} Nov 03 17:15:05.893825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.893804 6001 generic.go:356] "Generic (PLEG): container finished" podID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerID="cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51" exitCode=60 Nov 03 17:15:05.893906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.893854 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerDied","Data":"cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51"} Nov 03 17:15:05.893947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.893908 6001 scope.go:122] "RemoveContainer" containerID="285f2c7a435e5d48e83671333371bca1152aa23c31995d8a66fc18e7fa9268c4" Nov 03 17:15:05.894152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.894132 6001 scope.go:122] "RemoveContainer" containerID="cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51" Nov 03 17:15:05.894790 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:05.894764 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc_odh-ai-gateway-infra(6a4eae88-7286-4d3f-9032-9984cf11eefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" Nov 03 17:15:05.896215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.896194 6001 generic.go:356] "Generic (PLEG): container finished" podID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerID="e3e0b0dd6f63d4024747b8ef15487ef95e094df98225641464a0fb70ec08f22f" exitCode=60 Nov 03 17:15:05.896304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.896277 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerDied","Data":"e3e0b0dd6f63d4024747b8ef15487ef95e094df98225641464a0fb70ec08f22f"} Nov 03 17:15:05.898041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.898022 6001 generic.go:356] "Generic (PLEG): container finished" podID="9325a012-946b-4027-b2b6-dbb03b14b915" containerID="741cc1b2a22390270b35ab420ee6d33afcb4584468dc398ceea03301b20b750c" exitCode=60 Nov 03 17:15:05.898138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.898102 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerDied","Data":"741cc1b2a22390270b35ab420ee6d33afcb4584468dc398ceea03301b20b750c"} Nov 03 17:15:05.898318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.898300 6001 scope.go:122] "RemoveContainer" containerID="741cc1b2a22390270b35ab420ee6d33afcb4584468dc398ceea03301b20b750c" Nov 03 17:15:05.899979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.899961 6001 generic.go:356] "Generic (PLEG): container finished" podID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerID="3e025eb18001b5094ae5e825f2e53239dfe5be9438bae7042f9c0d1ca2c692df" exitCode=60 Nov 03 17:15:05.900094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:05.899998 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerDied","Data":"3e025eb18001b5094ae5e825f2e53239dfe5be9438bae7042f9c0d1ca2c692df"} Nov 03 17:15:06.042677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.042643 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68"] Nov 03 17:15:06.060780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.060749 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-099f2358-drb68"] Nov 03 17:15:06.070591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.069614 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" podStartSLOduration=6.069593972 podStartE2EDuration="6.069593972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:06.06714001 +0000 UTC m=+3601.400353583" watchObservedRunningTime="2026-11-03 17:15:06.069593972 +0000 UTC m=+3601.402807542" Nov 03 17:15:06.859501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.859463 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babc1056-32ae-47ed-9283-d70d59db4df5" path="/var/lib/kubelet/pods/babc1056-32ae-47ed-9283-d70d59db4df5/volumes" Nov 03 17:15:06.860248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.860227 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60" path="/var/lib/kubelet/pods/d6e428e1-d8d9-495e-a8d5-4c56d3c5aa60/volumes" Nov 03 17:15:06.915178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.915148 6001 generic.go:356] "Generic (PLEG): container finished" podID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerID="fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687" exitCode=60 Nov 03 17:15:06.915610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.915233 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerDied","Data":"fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687"} Nov 03 17:15:06.915610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.915299 6001 scope.go:122] "RemoveContainer" containerID="f0a4bb4a48608128811eb97c527449b44a8a5a4ff486d76a8ba7638e2aefe577" Nov 03 17:15:06.915734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.915623 6001 scope.go:122] "RemoveContainer" containerID="fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687" Nov 03 17:15:06.916113 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.916084 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c_odh-ai-gateway-infra(a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" Nov 03 17:15:06.917890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.917857 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerID="3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97" exitCode=60 Nov 03 17:15:06.917987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.917940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerDied","Data":"3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97"} Nov 03 17:15:06.918247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.918233 6001 scope.go:122] "RemoveContainer" containerID="3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97" Nov 03 17:15:06.918685 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.918661 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h_odh-ai-gateway-infra(9a14583c-2fca-4c77-b73b-8a84f18e2c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" Nov 03 17:15:06.920281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.920262 6001 scope.go:122] "RemoveContainer" containerID="cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51" Nov 03 17:15:06.920756 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.920730 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc_odh-ai-gateway-infra(6a4eae88-7286-4d3f-9032-9984cf11eefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" Nov 03 17:15:06.922118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.922099 6001 generic.go:356] "Generic (PLEG): container finished" podID="90629a0b-74c3-477e-9468-72681dd5b78d" containerID="09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043" exitCode=60 Nov 03 17:15:06.922212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.922177 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerDied","Data":"09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043"} Nov 03 17:15:06.922571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.922550 6001 scope.go:122] "RemoveContainer" containerID="09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043" Nov 03 17:15:06.922959 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.922936 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx_odh-ai-gateway-infra(90629a0b-74c3-477e-9468-72681dd5b78d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" Nov 03 17:15:06.924417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.924398 6001 generic.go:356] "Generic (PLEG): container finished" podID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerID="c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1" exitCode=60 Nov 03 17:15:06.924542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.924494 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerDied","Data":"c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1"} Nov 03 17:15:06.924798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.924785 6001 scope.go:122] "RemoveContainer" containerID="c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1" Nov 03 17:15:06.925196 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.925171 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf_odh-ai-gateway-infra(b7c64aac-55d0-4f28-8827-fa11441e5481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" Nov 03 17:15:06.926641 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.926622 6001 generic.go:356] "Generic (PLEG): container finished" podID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerID="71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3" exitCode=60 Nov 03 17:15:06.926729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.926677 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerDied","Data":"71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3"} Nov 03 17:15:06.927005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.926989 6001 scope.go:122] "RemoveContainer" containerID="71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3" Nov 03 17:15:06.927342 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.927321 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6_odh-ai-gateway-infra(510a81ce-d3b9-48f1-9a64-0912f32376df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" Nov 03 17:15:06.928807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.928789 6001 generic.go:356] "Generic (PLEG): container finished" podID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerID="10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a" exitCode=60 Nov 03 17:15:06.928886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.928857 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerDied","Data":"10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a"} Nov 03 17:15:06.929149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.929135 6001 scope.go:122] "RemoveContainer" containerID="10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a" Nov 03 17:15:06.929525 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.929503 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9_odh-ai-gateway-infra(b91179a4-e019-4998-b071-736a3c0f4ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" Nov 03 17:15:06.931105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.931082 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerStarted","Data":"f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d"} Nov 03 17:15:06.933499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.933481 6001 generic.go:356] "Generic (PLEG): container finished" podID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerID="674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b" exitCode=60 Nov 03 17:15:06.933595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.933532 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerDied","Data":"674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b"} Nov 03 17:15:06.933842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.933816 6001 scope.go:122] "RemoveContainer" containerID="674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b" Nov 03 17:15:06.934158 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.934133 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7_odh-ai-gateway-infra(5bcd7369-352d-4d61-a346-7004aee7d0ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" Nov 03 17:15:06.935618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.935597 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerID="a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e" exitCode=60 Nov 03 17:15:06.935717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.935677 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerDied","Data":"a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e"} Nov 03 17:15:06.935942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.935928 6001 scope.go:122] "RemoveContainer" containerID="a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e" Nov 03 17:15:06.936279 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.936255 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx_odh-ai-gateway-infra(1c2d8a27-b230-4a8b-ab7e-498e610069ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" Nov 03 17:15:06.937699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.937678 6001 generic.go:356] "Generic (PLEG): container finished" podID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerID="ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b" exitCode=60 Nov 03 17:15:06.937790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.937742 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerDied","Data":"ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b"} Nov 03 17:15:06.938039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.938023 6001 scope.go:122] "RemoveContainer" containerID="ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b" Nov 03 17:15:06.938352 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.938333 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-m6l29_odh-ai-gateway-infra(b9b61654-878c-4b35-9b1f-e25bb66698ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" Nov 03 17:15:06.939808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.939789 6001 generic.go:356] "Generic (PLEG): container finished" podID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerID="14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713" exitCode=60 Nov 03 17:15:06.939902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.939876 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerDied","Data":"14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713"} Nov 03 17:15:06.940136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.940118 6001 scope.go:122] "RemoveContainer" containerID="14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713" Nov 03 17:15:06.940497 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.940474 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn_odh-ai-gateway-infra(bb33eb94-227c-4d5c-bf79-c6692cd369a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" Nov 03 17:15:06.941974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.941951 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerStarted","Data":"b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee"} Nov 03 17:15:06.944144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.944117 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerStarted","Data":"b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd"} Nov 03 17:15:06.948008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.947793 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerDied","Data":"99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624"} Nov 03 17:15:06.948096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.947513 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerID="99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624" exitCode=60 Nov 03 17:15:06.949223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.948427 6001 scope.go:122] "RemoveContainer" containerID="99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624" Nov 03 17:15:06.949223 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.948960 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm_odh-ai-gateway-infra(1e9b5cfb-83ef-409a-9ad5-641495c9a860)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" Nov 03 17:15:06.952431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.952404 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerID="6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163" exitCode=60 Nov 03 17:15:06.952569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.952514 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerDied","Data":"6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163"} Nov 03 17:15:06.952992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.952969 6001 scope.go:122] "RemoveContainer" containerID="6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163" Nov 03 17:15:06.953385 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:06.953352 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755_odh-ai-gateway-infra(2ad571d0-2095-4b3a-9c41-bd4b393fdbfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" Nov 03 17:15:06.956706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.956030 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerStarted","Data":"a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861"} Nov 03 17:15:06.959942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:06.959912 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerStarted","Data":"69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c"} Nov 03 17:15:07.037611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.037574 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh"] Nov 03 17:15:07.043868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.043833 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af367c84-fpmwh"] Nov 03 17:15:07.441320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.441261 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" podStartSLOduration=7.441244946 podStartE2EDuration="7.441244946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:07.435979494 +0000 UTC m=+3602.769193064" watchObservedRunningTime="2026-11-03 17:15:07.441244946 +0000 UTC m=+3602.774458518" Nov 03 17:15:07.525942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.525846 6001 scope.go:122] "RemoveContainer" containerID="00f4b2989d3a1afab49195ee12a2e3ddeac809b7bf63a680cb5bfee6b26f640b" Nov 03 17:15:07.543116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.543093 6001 scope.go:122] "RemoveContainer" containerID="6414734145bf2dc39d92970840d81061159bbd13781db8b1e20c28fd97355c51" Nov 03 17:15:07.560926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.560873 6001 scope.go:122] "RemoveContainer" containerID="307bd76d903dea5f6d8f13a211252d5cd691e53a582d1aeff61cd6469fbcca25" Nov 03 17:15:07.578710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.578681 6001 scope.go:122] "RemoveContainer" containerID="6d555ba5f2fef00616952175a5517892307542241ebf332580159a07d6003435" Nov 03 17:15:07.595484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.595462 6001 scope.go:122] "RemoveContainer" containerID="e3e0b0dd6f63d4024747b8ef15487ef95e094df98225641464a0fb70ec08f22f" Nov 03 17:15:07.612818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.612790 6001 scope.go:122] "RemoveContainer" containerID="21c93e7872d93355ec6328cdcba1c35b1702fb21293b13d2b9413517250f288d" Nov 03 17:15:07.630429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.630403 6001 scope.go:122] "RemoveContainer" containerID="204327f1ee56745cc5170c781681081759ab87aa741e7bf746d205716ad1ff67" Nov 03 17:15:07.649399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.649372 6001 scope.go:122] "RemoveContainer" containerID="49571fc4becc4e11b2ba8274fc5e58fed01abe892f7683bb9b90de1291f6aa3e" Nov 03 17:15:07.666628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.666598 6001 scope.go:122] "RemoveContainer" containerID="3e025eb18001b5094ae5e825f2e53239dfe5be9438bae7042f9c0d1ca2c692df" Nov 03 17:15:07.683327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.683284 6001 scope.go:122] "RemoveContainer" containerID="d6547bdcab49ef95cd33bd44c0d8ea231e94466a73df1a38d76d297664970f8b" Nov 03 17:15:07.706163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.706135 6001 scope.go:122] "RemoveContainer" containerID="97224ae66339ad4ea5ee1500765b211e8cb2ba2a24c3c57f7ca543b3bdc99f2e" Nov 03 17:15:07.977381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.977338 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerID="a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861" exitCode=60 Nov 03 17:15:07.977936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.977418 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerDied","Data":"a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861"} Nov 03 17:15:07.977936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.977520 6001 scope.go:122] "RemoveContainer" containerID="147c3ce3f6caf99eb65a3e911bebe44094f1365f819fad4739c35d1de060d663" Nov 03 17:15:07.978085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.978060 6001 scope.go:122] "RemoveContainer" containerID="a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861" Nov 03 17:15:07.978570 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.978543 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q_odh-ai-gateway-infra(9a601ed0-5b71-42a3-8974-5d7c0770ffb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" Nov 03 17:15:07.980409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.980382 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbe06000-58db-4eeb-86ff-1e604357086c" containerID="69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c" exitCode=60 Nov 03 17:15:07.980551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.980492 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerDied","Data":"69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c"} Nov 03 17:15:07.980851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.980836 6001 scope.go:122] "RemoveContainer" containerID="69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c" Nov 03 17:15:07.981372 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.981349 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7_odh-ai-gateway-infra(dbe06000-58db-4eeb-86ff-1e604357086c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" Nov 03 17:15:07.983152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.983131 6001 scope.go:122] "RemoveContainer" containerID="fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687" Nov 03 17:15:07.983786 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.983763 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c_odh-ai-gateway-infra(a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" Nov 03 17:15:07.985204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.985185 6001 scope.go:122] "RemoveContainer" containerID="3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97" Nov 03 17:15:07.985667 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.985645 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h_odh-ai-gateway-infra(9a14583c-2fca-4c77-b73b-8a84f18e2c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" Nov 03 17:15:07.987241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.987224 6001 scope.go:122] "RemoveContainer" containerID="09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043" Nov 03 17:15:07.987711 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.987683 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx_odh-ai-gateway-infra(90629a0b-74c3-477e-9468-72681dd5b78d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" Nov 03 17:15:07.989242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.989220 6001 scope.go:122] "RemoveContainer" containerID="c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1" Nov 03 17:15:07.989675 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.989650 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf_odh-ai-gateway-infra(b7c64aac-55d0-4f28-8827-fa11441e5481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" Nov 03 17:15:07.991209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.991191 6001 scope.go:122] "RemoveContainer" containerID="71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3" Nov 03 17:15:07.991584 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.991561 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6_odh-ai-gateway-infra(510a81ce-d3b9-48f1-9a64-0912f32376df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" Nov 03 17:15:07.993404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.993384 6001 scope.go:122] "RemoveContainer" containerID="10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a" Nov 03 17:15:07.993842 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.993814 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9_odh-ai-gateway-infra(b91179a4-e019-4998-b071-736a3c0f4ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" Nov 03 17:15:07.995372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.995349 6001 generic.go:356] "Generic (PLEG): container finished" podID="9325a012-946b-4027-b2b6-dbb03b14b915" containerID="f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d" exitCode=60 Nov 03 17:15:07.995511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.995384 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerDied","Data":"f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d"} Nov 03 17:15:07.995714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.995697 6001 scope.go:122] "RemoveContainer" containerID="f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d" Nov 03 17:15:07.996050 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.996027 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6_odh-ai-gateway-infra(9325a012-946b-4027-b2b6-dbb03b14b915)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" Nov 03 17:15:07.997120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.997102 6001 scope.go:122] "RemoveContainer" containerID="b5dde2b8ee63723925f7ea73a4cceca5cca60245942b6a2d63273b161d7185ed" Nov 03 17:15:07.997852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.997833 6001 scope.go:122] "RemoveContainer" containerID="674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b" Nov 03 17:15:07.998201 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:07.998180 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7_odh-ai-gateway-infra(5bcd7369-352d-4d61-a346-7004aee7d0ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" Nov 03 17:15:07.999821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:07.999801 6001 scope.go:122] "RemoveContainer" containerID="a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e" Nov 03 17:15:08.000166 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.000140 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx_odh-ai-gateway-infra(1c2d8a27-b230-4a8b-ab7e-498e610069ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" Nov 03 17:15:08.001567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.001550 6001 scope.go:122] "RemoveContainer" containerID="ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b" Nov 03 17:15:08.001872 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.001852 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-m6l29_odh-ai-gateway-infra(b9b61654-878c-4b35-9b1f-e25bb66698ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" Nov 03 17:15:08.003308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.003290 6001 scope.go:122] "RemoveContainer" containerID="14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713" Nov 03 17:15:08.003639 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.003617 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn_odh-ai-gateway-infra(bb33eb94-227c-4d5c-bf79-c6692cd369a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" Nov 03 17:15:08.005183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.005161 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fcb5379-f832-449f-827e-acce7c4c3285" containerID="b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee" exitCode=60 Nov 03 17:15:08.005280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.005229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerDied","Data":"b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee"} Nov 03 17:15:08.005455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.005428 6001 scope.go:122] "RemoveContainer" containerID="b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee" Nov 03 17:15:08.005796 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.005774 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx_odh-ai-gateway-infra(3fcb5379-f832-449f-827e-acce7c4c3285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" Nov 03 17:15:08.007324 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.007304 6001 generic.go:356] "Generic (PLEG): container finished" podID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerID="b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd" exitCode=60 Nov 03 17:15:08.007431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.007381 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerDied","Data":"b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd"} Nov 03 17:15:08.007605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.007590 6001 scope.go:122] "RemoveContainer" containerID="b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd" Nov 03 17:15:08.007898 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.007875 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7_odh-ai-gateway-infra(4ee7480b-8265-4acb-b9e3-b61dfb2b6cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" Nov 03 17:15:08.009534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.009517 6001 scope.go:122] "RemoveContainer" containerID="99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624" Nov 03 17:15:08.009841 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.009819 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm_odh-ai-gateway-infra(1e9b5cfb-83ef-409a-9ad5-641495c9a860)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" Nov 03 17:15:08.011250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.011234 6001 scope.go:122] "RemoveContainer" containerID="6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163" Nov 03 17:15:08.011517 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:08.011500 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755_odh-ai-gateway-infra(2ad571d0-2095-4b3a-9c41-bd4b393fdbfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" Nov 03 17:15:08.014977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.014959 6001 scope.go:122] "RemoveContainer" containerID="741cc1b2a22390270b35ab420ee6d33afcb4584468dc398ceea03301b20b750c" Nov 03 17:15:08.044526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.044501 6001 scope.go:122] "RemoveContainer" containerID="4c1f8c8af8a3ed5993d9b510c952b6d4e2fd607c311a3b7deb3e4594d5346d7e" Nov 03 17:15:08.063880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.063858 6001 scope.go:122] "RemoveContainer" containerID="e5721ecedcf6bee6796651d3c29415d3f11b3a3da9d6ef3fbbb09a8b239eeb62" Nov 03 17:15:08.875467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:08.874865 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e426c3-2da4-46b9-8f17-41103b42abbd" path="/var/lib/kubelet/pods/99e426c3-2da4-46b9-8f17-41103b42abbd/volumes" Nov 03 17:15:09.023413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:09.023383 6001 scope.go:122] "RemoveContainer" containerID="f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d" Nov 03 17:15:09.023954 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:09.023927 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6_odh-ai-gateway-infra(9325a012-946b-4027-b2b6-dbb03b14b915)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" Nov 03 17:15:09.025270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:09.025247 6001 scope.go:122] "RemoveContainer" containerID="b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee" Nov 03 17:15:09.025727 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:09.025703 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx_odh-ai-gateway-infra(3fcb5379-f832-449f-827e-acce7c4c3285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" Nov 03 17:15:09.027085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:09.027066 6001 scope.go:122] "RemoveContainer" containerID="b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd" Nov 03 17:15:09.027550 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:09.027519 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7_odh-ai-gateway-infra(4ee7480b-8265-4acb-b9e3-b61dfb2b6cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" Nov 03 17:15:09.028775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:09.028757 6001 scope.go:122] "RemoveContainer" containerID="a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861" Nov 03 17:15:09.029110 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:09.029084 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q_odh-ai-gateway-infra(9a601ed0-5b71-42a3-8974-5d7c0770ffb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" Nov 03 17:15:09.030519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:09.030500 6001 scope.go:122] "RemoveContainer" containerID="69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c" Nov 03 17:15:09.030748 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:09.030728 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7_odh-ai-gateway-infra(dbe06000-58db-4eeb-86ff-1e604357086c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" Nov 03 17:15:10.512295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.512252 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:15:10.518115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.518090 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:10.533886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.533856 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cb0fa3\"" Nov 03 17:15:10.536747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.536722 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:15:10.542384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.542337 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cb0fa3-dockercfg-zcgfx\"" Nov 03 17:15:10.913533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.906618 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:10.913533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:10.907257 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvp94\" (UniqueName: \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.022019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.021985 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvp94\" (UniqueName: \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.022253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.022093 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.022969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.022943 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.023107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.023074 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:15:11.030280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.030251 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:15:11.030482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.030349 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.047555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.047521 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvp94\" (UniqueName: \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\") pod \"payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.124318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.124275 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98cnb\" (UniqueName: \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.124558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.124367 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.136978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.136943 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:11.226661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.226627 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98cnb\" (UniqueName: \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.226921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.226666 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.227628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.227573 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.251904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.251872 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98cnb\" (UniqueName: \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\") pod \"payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.348429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.348393 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:11.449487 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:11.449386 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904551aa_b8b4_479e_b24b_247fec59c56d.slice/crio-2808121bb56fd23afa38b8e78994754c42d512c816fc186235aacf2d38e32a41 WatchSource:0}: Error finding container 2808121bb56fd23afa38b8e78994754c42d512c816fc186235aacf2d38e32a41: Status 404 returned error can't find the container with id 2808121bb56fd23afa38b8e78994754c42d512c816fc186235aacf2d38e32a41 Nov 03 17:15:11.454910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.454877 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:15:11.661130 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:15:11.661097 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981fa68e_4312_47f6_8f40_8e96d706b102.slice/crio-9da725249a3aa89f599d0ca15e990ca74d8234f52c4773ffa001acdbae14237e WatchSource:0}: Error finding container 9da725249a3aa89f599d0ca15e990ca74d8234f52c4773ffa001acdbae14237e: Status 404 returned error can't find the container with id 9da725249a3aa89f599d0ca15e990ca74d8234f52c4773ffa001acdbae14237e Nov 03 17:15:11.664235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.664213 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:15:11.723851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.723820 6001 scope.go:122] "RemoveContainer" containerID="5b5964c0fae80ee3fe6cd8fbe86de53dd55b0f3bee600430d9ef681d21debf71" Nov 03 17:15:11.742115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.742076 6001 scope.go:122] "RemoveContainer" containerID="23b29ea0fb45e23078e63d21cd58a1a4d58c19849a50e888ab2fcaf1005e31c4" Nov 03 17:15:11.755436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.755410 6001 scope.go:122] "RemoveContainer" containerID="61e1005d56637ef30eb897ed86722845ceeb4113d8e9d5ae215ff7a2964e8130" Nov 03 17:15:11.767788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.767765 6001 scope.go:122] "RemoveContainer" containerID="27e077b0d6a695c79cf776a7c860d6c0a03917d288e057fe2e0cab5cb264e5ea" Nov 03 17:15:11.780132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:11.780102 6001 scope.go:122] "RemoveContainer" containerID="b6efbd876ecb42b3eec8afcbefa4ca8d043b83771087996b5d97f74f5bd28154" Nov 03 17:15:12.070269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.070229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" event={"ID":"981fa68e-4312-47f6-8f40-8e96d706b102","Type":"ContainerStarted","Data":"0968bc25fdae5074dad7ba8f5456b6387c23d39bc9b072126d8136ed6ac4d24d"} Nov 03 17:15:12.070495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.070294 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" event={"ID":"981fa68e-4312-47f6-8f40-8e96d706b102","Type":"ContainerStarted","Data":"9da725249a3aa89f599d0ca15e990ca74d8234f52c4773ffa001acdbae14237e"} Nov 03 17:15:12.070495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.070324 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:12.072515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.072480 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" event={"ID":"904551aa-b8b4-479e-b24b-247fec59c56d","Type":"ContainerStarted","Data":"61d9bf927879183c2435ceda76a16629c04beea925a964a38631d371809b973a"} Nov 03 17:15:12.072727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.072544 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" event={"ID":"904551aa-b8b4-479e-b24b-247fec59c56d","Type":"ContainerStarted","Data":"2808121bb56fd23afa38b8e78994754c42d512c816fc186235aacf2d38e32a41"} Nov 03 17:15:12.072727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.072580 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:12.157529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.153417 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" podStartSLOduration=2.15339291 podStartE2EDuration="2.15339291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:12.125884223 +0000 UTC m=+3607.459097804" watchObservedRunningTime="2026-11-03 17:15:12.15339291 +0000 UTC m=+3607.486606492" Nov 03 17:15:12.194672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:12.194599 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" podStartSLOduration=2.194573289 podStartE2EDuration="2.194573289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:12.189690131 +0000 UTC m=+3607.522903700" watchObservedRunningTime="2026-11-03 17:15:12.194573289 +0000 UTC m=+3607.527786859" Nov 03 17:15:16.035616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:16.035582 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj"] Nov 03 17:15:16.048830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:16.048799 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd0fe45b-zpwmj"] Nov 03 17:15:16.794520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:16.794482 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41669dd-d2d5-414f-be11-a53541443ae6" path="/var/lib/kubelet/pods/a41669dd-d2d5-414f-be11-a53541443ae6/volumes" Nov 03 17:15:17.781771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:17.781736 6001 scope.go:122] "RemoveContainer" containerID="09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043" Nov 03 17:15:17.782264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:17.781953 6001 scope.go:122] "RemoveContainer" containerID="c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1" Nov 03 17:15:17.782264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:17.782149 6001 scope.go:122] "RemoveContainer" containerID="674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b" Nov 03 17:15:17.782405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:17.782323 6001 scope.go:122] "RemoveContainer" containerID="cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51" Nov 03 17:15:18.782222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:18.782178 6001 scope.go:122] "RemoveContainer" containerID="b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee" Nov 03 17:15:18.782681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:18.782434 6001 scope.go:122] "RemoveContainer" containerID="14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713" Nov 03 17:15:18.782864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:18.782771 6001 scope.go:122] "RemoveContainer" containerID="fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687" Nov 03 17:15:18.783070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:18.783041 6001 scope.go:122] "RemoveContainer" containerID="99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624" Nov 03 17:15:19.161266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.161233 6001 generic.go:356] "Generic (PLEG): container finished" podID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerID="1d1f221158b468db7c372ab5ddef8ff7f60c62db71e112c34ab975a9dfc46fc5" exitCode=60 Nov 03 17:15:19.161433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.161283 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerDied","Data":"1d1f221158b468db7c372ab5ddef8ff7f60c62db71e112c34ab975a9dfc46fc5"} Nov 03 17:15:19.161433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.161351 6001 scope.go:122] "RemoveContainer" containerID="674bede127f7aa5863f6a90d81950697b8faaa479c716929ede8983a9f1f989b" Nov 03 17:15:19.161972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.161954 6001 scope.go:122] "RemoveContainer" containerID="1d1f221158b468db7c372ab5ddef8ff7f60c62db71e112c34ab975a9dfc46fc5" Nov 03 17:15:19.162514 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:19.162457 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7_odh-ai-gateway-infra(5bcd7369-352d-4d61-a346-7004aee7d0ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" Nov 03 17:15:19.164125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.164099 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerStarted","Data":"ea32117ac8ec96e56eb633c8b119c5544e0fc8e917f970e7b55bc3cbfa1b7d97"} Nov 03 17:15:19.166661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.166634 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerStarted","Data":"a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d"} Nov 03 17:15:19.167123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.167103 6001 scope.go:122] "RemoveContainer" containerID="a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d" Nov 03 17:15:19.167581 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:19.167552 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx_odh-ai-gateway-infra(90629a0b-74c3-477e-9468-72681dd5b78d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" Nov 03 17:15:19.169141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.169116 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerStarted","Data":"7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57"} Nov 03 17:15:19.169552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.169535 6001 scope.go:122] "RemoveContainer" containerID="7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57" Nov 03 17:15:19.169912 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:19.169893 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf_odh-ai-gateway-infra(b7c64aac-55d0-4f28-8827-fa11441e5481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" Nov 03 17:15:19.782219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.782186 6001 scope.go:122] "RemoveContainer" containerID="6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163" Nov 03 17:15:19.782397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.782379 6001 scope.go:122] "RemoveContainer" containerID="10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a" Nov 03 17:15:19.782804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.782618 6001 scope.go:122] "RemoveContainer" containerID="ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b" Nov 03 17:15:19.783104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:19.783056 6001 scope.go:122] "RemoveContainer" containerID="69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c" Nov 03 17:15:20.186944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.186913 6001 generic.go:356] "Generic (PLEG): container finished" podID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerID="ea32117ac8ec96e56eb633c8b119c5544e0fc8e917f970e7b55bc3cbfa1b7d97" exitCode=60 Nov 03 17:15:20.187111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.186988 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerDied","Data":"ea32117ac8ec96e56eb633c8b119c5544e0fc8e917f970e7b55bc3cbfa1b7d97"} Nov 03 17:15:20.187111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.187072 6001 scope.go:122] "RemoveContainer" containerID="cbb9a41197f71dcc4c2eef216aa0b3436854ea35c2ec75d15abb20d01ec40f51" Nov 03 17:15:20.187636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.187621 6001 scope.go:122] "RemoveContainer" containerID="ea32117ac8ec96e56eb633c8b119c5544e0fc8e917f970e7b55bc3cbfa1b7d97" Nov 03 17:15:20.188182 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.188160 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc_odh-ai-gateway-infra(6a4eae88-7286-4d3f-9032-9984cf11eefa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" Nov 03 17:15:20.190092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.190070 6001 generic.go:356] "Generic (PLEG): container finished" podID="90629a0b-74c3-477e-9468-72681dd5b78d" containerID="a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d" exitCode=60 Nov 03 17:15:20.190196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.190144 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerDied","Data":"a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d"} Nov 03 17:15:20.190740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.190721 6001 scope.go:122] "RemoveContainer" containerID="a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d" Nov 03 17:15:20.191197 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.191178 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx_odh-ai-gateway-infra(90629a0b-74c3-477e-9468-72681dd5b78d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" Nov 03 17:15:20.192638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.192610 6001 generic.go:356] "Generic (PLEG): container finished" podID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerID="7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57" exitCode=60 Nov 03 17:15:20.192938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.192910 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerDied","Data":"7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57"} Nov 03 17:15:20.193368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.193350 6001 scope.go:122] "RemoveContainer" containerID="7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57" Nov 03 17:15:20.193889 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.193865 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf_odh-ai-gateway-infra(b7c64aac-55d0-4f28-8827-fa11441e5481)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" Nov 03 17:15:20.199794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.199755 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerStarted","Data":"4bb82d012f12b442f6c49ce94ab0ef1ac4cd22abd4567d6fca917825e72b6f59"} Nov 03 17:15:20.206128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.206091 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fcb5379-f832-449f-827e-acce7c4c3285" containerID="af6953e13084a43a21d1dd79ac958beba64436423c0bdf59fccb20bfcf205d8a" exitCode=60 Nov 03 17:15:20.206321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.206189 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerDied","Data":"af6953e13084a43a21d1dd79ac958beba64436423c0bdf59fccb20bfcf205d8a"} Nov 03 17:15:20.207307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.206765 6001 scope.go:122] "RemoveContainer" containerID="af6953e13084a43a21d1dd79ac958beba64436423c0bdf59fccb20bfcf205d8a" Nov 03 17:15:20.207307 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.207249 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx_odh-ai-gateway-infra(3fcb5379-f832-449f-827e-acce7c4c3285)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" Nov 03 17:15:20.213113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.213090 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerID="694049b2d1ec8a74749a8596ed3774a0effa650f1199332fa21ed486ba96c5d6" exitCode=60 Nov 03 17:15:20.213248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.213235 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerDied","Data":"694049b2d1ec8a74749a8596ed3774a0effa650f1199332fa21ed486ba96c5d6"} Nov 03 17:15:20.213703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.213679 6001 scope.go:122] "RemoveContainer" containerID="694049b2d1ec8a74749a8596ed3774a0effa650f1199332fa21ed486ba96c5d6" Nov 03 17:15:20.214013 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.213993 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm_odh-ai-gateway-infra(1e9b5cfb-83ef-409a-9ad5-641495c9a860)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" Nov 03 17:15:20.216395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.216370 6001 generic.go:356] "Generic (PLEG): container finished" podID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerID="d842d0a06b1c745615af7bb2ec2da7400d1d29189972aaf9c625beb4ad971180" exitCode=60 Nov 03 17:15:20.216597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.216577 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerDied","Data":"d842d0a06b1c745615af7bb2ec2da7400d1d29189972aaf9c625beb4ad971180"} Nov 03 17:15:20.217145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.217122 6001 scope.go:122] "RemoveContainer" containerID="d842d0a06b1c745615af7bb2ec2da7400d1d29189972aaf9c625beb4ad971180" Nov 03 17:15:20.217556 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:20.217531 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c_odh-ai-gateway-infra(a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" Nov 03 17:15:20.233318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.233275 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:20.273112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.273075 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:20.397556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.397528 6001 scope.go:122] "RemoveContainer" containerID="09f795866d816d4b22eaf612aaa5d12c3cfb249d7ce692f0972dc0c6cfbec043" Nov 03 17:15:20.409392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.409358 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:20.447491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.447416 6001 scope.go:122] "RemoveContainer" containerID="c3ac3ff2bc6fe70f8af1eaf9d3fa35db8ecf2cb2f6b337225771ebc3e05ed3c1" Nov 03 17:15:20.459232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.459185 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" podStartSLOduration=20.459168577 podStartE2EDuration="20.459168577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:20.369854737 +0000 UTC m=+3615.703068331" watchObservedRunningTime="2026-11-03 17:15:20.459168577 +0000 UTC m=+3615.792382145" Nov 03 17:15:20.467028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.466970 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:20.474846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.474769 6001 scope.go:122] "RemoveContainer" containerID="b4d5f52edf3b81cc174322876e78c020606daf31e60859c3e860cb7073e793ee" Nov 03 17:15:20.499265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.499235 6001 scope.go:122] "RemoveContainer" containerID="99fef8456e18a66231b95e0816ba261905005a1e7d230bc8487138fd58bc4624" Nov 03 17:15:20.557356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.557334 6001 scope.go:122] "RemoveContainer" containerID="fef01b3594957602b5b7f68b3c0356cad48cf6356f84d3fd75d50425d07c6687" Nov 03 17:15:20.565414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.565330 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:20.671541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.660914 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\") pod \"5bcd7369-352d-4d61-a346-7004aee7d0ef\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " Nov 03 17:15:20.671541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.660982 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\" (UniqueName: \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\") pod \"5bcd7369-352d-4d61-a346-7004aee7d0ef\" (UID: \"5bcd7369-352d-4d61-a346-7004aee7d0ef\") " Nov 03 17:15:20.671541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.661320 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca" pod "5bcd7369-352d-4d61-a346-7004aee7d0ef" (UID: "5bcd7369-352d-4d61-a346-7004aee7d0ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:20.671541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.662123 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcd7369-352d-4d61-a346-7004aee7d0ef-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.671541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.669984 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw" pod "5bcd7369-352d-4d61-a346-7004aee7d0ef" (UID: "5bcd7369-352d-4d61-a346-7004aee7d0ef"). InnerVolumeSpecName "kube-api-access-7c4jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:20.764637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.764596 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c4jw\" (UniqueName: \"kubernetes.io/projected/5bcd7369-352d-4d61-a346-7004aee7d0ef-kube-api-access-7c4jw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.781769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.781735 6001 scope.go:122] "RemoveContainer" containerID="f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d" Nov 03 17:15:20.782524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.782150 6001 scope.go:122] "RemoveContainer" containerID="b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd" Nov 03 17:15:20.782524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.782248 6001 scope.go:122] "RemoveContainer" containerID="3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97" Nov 03 17:15:20.783095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:20.782714 6001 scope.go:122] "RemoveContainer" containerID="71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3" Nov 03 17:15:21.033466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.033413 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25"] Nov 03 17:15:21.039619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.039579 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fd35899-sts25"] Nov 03 17:15:21.228797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.228766 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" Nov 03 17:15:21.229011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.228773 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7" event={"ID":"5bcd7369-352d-4d61-a346-7004aee7d0ef","Type":"ContainerDied","Data":"dfa5174c49e29ee2248220c6a0aa35ac583ff8466556cfe523e766d531564d00"} Nov 03 17:15:21.229011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.228896 6001 scope.go:122] "RemoveContainer" containerID="1d1f221158b468db7c372ab5ddef8ff7f60c62db71e112c34ab975a9dfc46fc5" Nov 03 17:15:21.231356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.231310 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerStarted","Data":"fc77c419ed40debcef3e7bcfbff702313a964d79ad80c07b0851dc5b3dd31c74"} Nov 03 17:15:21.233889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.233863 6001 generic.go:356] "Generic (PLEG): container finished" podID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerID="4bb82d012f12b442f6c49ce94ab0ef1ac4cd22abd4567d6fca917825e72b6f59" exitCode=60 Nov 03 17:15:21.234016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.233930 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerDied","Data":"4bb82d012f12b442f6c49ce94ab0ef1ac4cd22abd4567d6fca917825e72b6f59"} Nov 03 17:15:21.234612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.234575 6001 scope.go:122] "RemoveContainer" containerID="4bb82d012f12b442f6c49ce94ab0ef1ac4cd22abd4567d6fca917825e72b6f59" Nov 03 17:15:21.235091 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:21.235068 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn_odh-ai-gateway-infra(bb33eb94-227c-4d5c-bf79-c6692cd369a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" Nov 03 17:15:21.240581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.240527 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerID="0b34158a354821c550f22490db917e999415e51d50c71b264e636905ba0dd6ab" exitCode=60 Nov 03 17:15:21.240581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.240560 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerDied","Data":"0b34158a354821c550f22490db917e999415e51d50c71b264e636905ba0dd6ab"} Nov 03 17:15:21.241116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.241093 6001 scope.go:122] "RemoveContainer" containerID="0b34158a354821c550f22490db917e999415e51d50c71b264e636905ba0dd6ab" Nov 03 17:15:21.241617 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:21.241588 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755_odh-ai-gateway-infra(2ad571d0-2095-4b3a-9c41-bd4b393fdbfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" Nov 03 17:15:21.243131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.243110 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerStarted","Data":"4fe618d78ff21809bd6b5a0f3ff1cf8710d4802ae350cb759c34f54d7ed2e4ba"} Nov 03 17:15:21.254801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.254673 6001 scope.go:122] "RemoveContainer" containerID="14d7aab0e9c39467b1defdc0a5d46034e94501187ccafe979bc4b80378bad713" Nov 03 17:15:21.275268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.275234 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerStarted","Data":"c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da"} Nov 03 17:15:21.276162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.275832 6001 scope.go:122] "RemoveContainer" containerID="c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da" Nov 03 17:15:21.278454 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:21.276496 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9_odh-ai-gateway-infra(b91179a4-e019-4998-b071-736a3c0f4ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" Nov 03 17:15:21.309289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.309181 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:21.342396 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.342369 6001 scope.go:122] "RemoveContainer" containerID="6dcee76e9f1cdd884473f83ab9b671ada225e9dbd3906175ff883f7800ec0163" Nov 03 17:15:21.402348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.398859 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:21.432126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.432058 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" podStartSLOduration=21.432038106 podStartE2EDuration="21.432038106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:21.42936738 +0000 UTC m=+3616.762580973" watchObservedRunningTime="2026-11-03 17:15:21.432038106 +0000 UTC m=+3616.765251675" Nov 03 17:15:21.455178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.455119 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:21.606165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.606097 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:21.615805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.615743 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895435-sb7b7"] Nov 03 17:15:21.638794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.638760 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:21.713195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.713163 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:21.783626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.782963 6001 scope.go:122] "RemoveContainer" containerID="a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e" Nov 03 17:15:21.784338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.784313 6001 scope.go:122] "RemoveContainer" containerID="a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861" Nov 03 17:15:21.798586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.798549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\") pod \"6a4eae88-7286-4d3f-9032-9984cf11eefa\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " Nov 03 17:15:21.798998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.798969 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\" (UniqueName: \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\") pod \"6a4eae88-7286-4d3f-9032-9984cf11eefa\" (UID: \"6a4eae88-7286-4d3f-9032-9984cf11eefa\") " Nov 03 17:15:21.800304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.799000 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca" pod "6a4eae88-7286-4d3f-9032-9984cf11eefa" (UID: "6a4eae88-7286-4d3f-9032-9984cf11eefa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:21.879829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.879790 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t" pod "6a4eae88-7286-4d3f-9032-9984cf11eefa" (UID: "6a4eae88-7286-4d3f-9032-9984cf11eefa"). InnerVolumeSpecName "kube-api-access-pgb9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:21.903861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.902789 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a4eae88-7286-4d3f-9032-9984cf11eefa-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:21.903861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.902827 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgb9t\" (UniqueName: \"kubernetes.io/projected/6a4eae88-7286-4d3f-9032-9984cf11eefa-kube-api-access-pgb9t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:21.922323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.921872 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:21.939786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:21.938142 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:22.041104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.039651 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\") pod \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " Nov 03 17:15:22.041104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.039737 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\" (UniqueName: \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\") pod \"90629a0b-74c3-477e-9468-72681dd5b78d\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " Nov 03 17:15:22.041104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.039834 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\") pod \"90629a0b-74c3-477e-9468-72681dd5b78d\" (UID: \"90629a0b-74c3-477e-9468-72681dd5b78d\") " Nov 03 17:15:22.041104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.039946 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\" (UniqueName: \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\") pod \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\" (UID: \"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7\") " Nov 03 17:15:22.061600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.048219 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca" pod "a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" (UID: "a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.065402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.065310 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5" pod "a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" (UID: "a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7"). InnerVolumeSpecName "kube-api-access-tpfw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.079917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.079710 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca" pod "90629a0b-74c3-477e-9468-72681dd5b78d" (UID: "90629a0b-74c3-477e-9468-72681dd5b78d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.101881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.100063 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4" pod "90629a0b-74c3-477e-9468-72681dd5b78d" (UID: "90629a0b-74c3-477e-9468-72681dd5b78d"). InnerVolumeSpecName "kube-api-access-s2tr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.123737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.121378 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:22.137832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.137800 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:22.146834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.146307 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\") pod \"b7c64aac-55d0-4f28-8827-fa11441e5481\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " Nov 03 17:15:22.146834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.146647 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\" (UniqueName: \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\") pod \"3fcb5379-f832-449f-827e-acce7c4c3285\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " Nov 03 17:15:22.148782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.148689 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca" pod "b7c64aac-55d0-4f28-8827-fa11441e5481" (UID: "b7c64aac-55d0-4f28-8827-fa11441e5481"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.149044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.149023 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2tr4\" (UniqueName: \"kubernetes.io/projected/90629a0b-74c3-477e-9468-72681dd5b78d-kube-api-access-s2tr4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.149185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.149169 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90629a0b-74c3-477e-9468-72681dd5b78d-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.149310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.149295 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpfw5\" (UniqueName: \"kubernetes.io/projected/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-kube-api-access-tpfw5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.149435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.149399 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c64aac-55d0-4f28-8827-fa11441e5481-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.149435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.149418 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.155345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.154989 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:22.251141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.250924 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\") pod \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " Nov 03 17:15:22.251141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.251039 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\") pod \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\" (UID: \"1e9b5cfb-83ef-409a-9ad5-641495c9a860\") " Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.251335 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\" (UniqueName: \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\") pod \"b7c64aac-55d0-4f28-8827-fa11441e5481\" (UID: \"b7c64aac-55d0-4f28-8827-fa11441e5481\") " Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.251381 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\") pod \"3fcb5379-f832-449f-827e-acce7c4c3285\" (UID: \"3fcb5379-f832-449f-827e-acce7c4c3285\") " Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.251715 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp" pod "3fcb5379-f832-449f-827e-acce7c4c3285" (UID: "3fcb5379-f832-449f-827e-acce7c4c3285"). InnerVolumeSpecName "kube-api-access-8h6tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.252233 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6tp\" (UniqueName: \"kubernetes.io/projected/3fcb5379-f832-449f-827e-acce7c4c3285-kube-api-access-8h6tp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.252686 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca" pod "3fcb5379-f832-449f-827e-acce7c4c3285" (UID: "3fcb5379-f832-449f-827e-acce7c4c3285"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.255597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.252713 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca" pod "1e9b5cfb-83ef-409a-9ad5-641495c9a860" (UID: "1e9b5cfb-83ef-409a-9ad5-641495c9a860"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.256358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.256060 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9" pod "1e9b5cfb-83ef-409a-9ad5-641495c9a860" (UID: "1e9b5cfb-83ef-409a-9ad5-641495c9a860"). InnerVolumeSpecName "kube-api-access-lwfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.260768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.260585 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj" pod "b7c64aac-55d0-4f28-8827-fa11441e5481" (UID: "b7c64aac-55d0-4f28-8827-fa11441e5481"). InnerVolumeSpecName "kube-api-access-446nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.293355 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.293318 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:15:22.308979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.308940 6001 generic.go:356] "Generic (PLEG): container finished" podID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerID="c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da" exitCode=60 Nov 03 17:15:22.309158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.309018 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerDied","Data":"c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da"} Nov 03 17:15:22.309158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.309100 6001 scope.go:122] "RemoveContainer" containerID="10ae1c2dc862d17cad663d19be358114a26601f9b5883adb74a8a757da8a099a" Nov 03 17:15:22.309789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.309707 6001 scope.go:122] "RemoveContainer" containerID="c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da" Nov 03 17:15:22.310240 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.310186 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9_odh-ai-gateway-infra(b91179a4-e019-4998-b071-736a3c0f4ab7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" Nov 03 17:15:22.311925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.311904 6001 generic.go:356] "Generic (PLEG): container finished" podID="9325a012-946b-4027-b2b6-dbb03b14b915" containerID="c57489903676cf372388f194b01ecfc054fa6aeb7deb24d33b78131c2bb3d0ae" exitCode=60 Nov 03 17:15:22.312026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.311979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerDied","Data":"c57489903676cf372388f194b01ecfc054fa6aeb7deb24d33b78131c2bb3d0ae"} Nov 03 17:15:22.312549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.312528 6001 scope.go:122] "RemoveContainer" containerID="c57489903676cf372388f194b01ecfc054fa6aeb7deb24d33b78131c2bb3d0ae" Nov 03 17:15:22.313044 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.313008 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6_odh-ai-gateway-infra(9325a012-946b-4027-b2b6-dbb03b14b915)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" Nov 03 17:15:22.315760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.315736 6001 generic.go:356] "Generic (PLEG): container finished" podID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerID="fc77c419ed40debcef3e7bcfbff702313a964d79ad80c07b0851dc5b3dd31c74" exitCode=60 Nov 03 17:15:22.315857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.315804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerDied","Data":"fc77c419ed40debcef3e7bcfbff702313a964d79ad80c07b0851dc5b3dd31c74"} Nov 03 17:15:22.316200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.316183 6001 scope.go:122] "RemoveContainer" containerID="fc77c419ed40debcef3e7bcfbff702313a964d79ad80c07b0851dc5b3dd31c74" Nov 03 17:15:22.316694 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.316654 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895435-m6l29_odh-ai-gateway-infra(b9b61654-878c-4b35-9b1f-e25bb66698ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" Nov 03 17:15:22.319962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.319944 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" Nov 03 17:15:22.320087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.319979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx" event={"ID":"3fcb5379-f832-449f-827e-acce7c4c3285","Type":"ContainerDied","Data":"0e5bb5ac1210b54168538482db19b5a5728117225ec82117f1df0b2ea158197c"} Nov 03 17:15:22.324004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.322991 6001 generic.go:356] "Generic (PLEG): container finished" podID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerID="a47a0789fb245f62befe90613e37fc5739246e24b8af5a2a4d0919a3f42454b4" exitCode=60 Nov 03 17:15:22.324004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.323066 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerDied","Data":"a47a0789fb245f62befe90613e37fc5739246e24b8af5a2a4d0919a3f42454b4"} Nov 03 17:15:22.324004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.323558 6001 scope.go:122] "RemoveContainer" containerID="a47a0789fb245f62befe90613e37fc5739246e24b8af5a2a4d0919a3f42454b4" Nov 03 17:15:22.324004 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.323937 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7_odh-ai-gateway-infra(4ee7480b-8265-4acb-b9e3-b61dfb2b6cef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" Nov 03 17:15:22.326548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.326110 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" event={"ID":"1e9b5cfb-83ef-409a-9ad5-641495c9a860","Type":"ContainerDied","Data":"9f4e25178d54f969b0bf24cacd0e578df6367b51826f8356f15f34554bf60f40"} Nov 03 17:15:22.326548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.326206 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm" Nov 03 17:15:22.353990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.353860 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:22.355251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.355209 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-446nj\" (UniqueName: \"kubernetes.io/projected/b7c64aac-55d0-4f28-8827-fa11441e5481-kube-api-access-446nj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.355376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.355254 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fcb5379-f832-449f-827e-acce7c4c3285-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.355376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.355270 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e9b5cfb-83ef-409a-9ad5-641495c9a860-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.355376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.355285 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfg9\" (UniqueName: \"kubernetes.io/projected/1e9b5cfb-83ef-409a-9ad5-641495c9a860-kube-api-access-lwfg9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.356702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.356683 6001 scope.go:122] "RemoveContainer" containerID="f719a7a6a630ecb37abaf5c3e4b65aa1ffcf70c62611cfdeb9b525764faead2d" Nov 03 17:15:22.363819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.363790 6001 generic.go:356] "Generic (PLEG): container finished" podID="dbe06000-58db-4eeb-86ff-1e604357086c" containerID="4fe618d78ff21809bd6b5a0f3ff1cf8710d4802ae350cb759c34f54d7ed2e4ba" exitCode=60 Nov 03 17:15:22.363968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.363948 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerDied","Data":"4fe618d78ff21809bd6b5a0f3ff1cf8710d4802ae350cb759c34f54d7ed2e4ba"} Nov 03 17:15:22.364413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.364391 6001 scope.go:122] "RemoveContainer" containerID="4fe618d78ff21809bd6b5a0f3ff1cf8710d4802ae350cb759c34f54d7ed2e4ba" Nov 03 17:15:22.364851 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.364825 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7_odh-ai-gateway-infra(dbe06000-58db-4eeb-86ff-1e604357086c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" Nov 03 17:15:22.369654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.366835 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" event={"ID":"a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7","Type":"ContainerDied","Data":"d5451045d565b6e384e79c12967a8588b58db85b7f983fee6b57ffc2135243ca"} Nov 03 17:15:22.369654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.366928 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c" Nov 03 17:15:22.380027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.379540 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerID="3ca0b2b29444613b67917380f8eb7aedabcfaed9d1b0b1847a3078d4ccdefcb9" exitCode=60 Nov 03 17:15:22.380027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.379618 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerDied","Data":"3ca0b2b29444613b67917380f8eb7aedabcfaed9d1b0b1847a3078d4ccdefcb9"} Nov 03 17:15:22.380817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.380352 6001 scope.go:122] "RemoveContainer" containerID="3ca0b2b29444613b67917380f8eb7aedabcfaed9d1b0b1847a3078d4ccdefcb9" Nov 03 17:15:22.380817 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:22.380763 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h_odh-ai-gateway-infra(9a14583c-2fca-4c77-b73b-8a84f18e2c43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" Nov 03 17:15:22.386243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.386216 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" event={"ID":"6a4eae88-7286-4d3f-9032-9984cf11eefa","Type":"ContainerDied","Data":"7225111f2bee4d1b4f6844437fa5b1b531a21e83f5c197c8294a94ebe5d3be1d"} Nov 03 17:15:22.389230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.389211 6001 scope.go:122] "RemoveContainer" containerID="ee7a6ecc3213edc5dd2bfb9d56d13844ad75b65ceec05707ca245170aad42a3b" Nov 03 17:15:22.394560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.394506 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc" Nov 03 17:15:22.398342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.397933 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:22.400180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.400110 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" event={"ID":"90629a0b-74c3-477e-9468-72681dd5b78d","Type":"ContainerDied","Data":"b71f47e76518338bff6e37549b44e7321e11153a4f4a3a640dee5ce5e699004b"} Nov 03 17:15:22.400279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.400196 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx" Nov 03 17:15:22.406848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.405493 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" event={"ID":"b7c64aac-55d0-4f28-8827-fa11441e5481","Type":"ContainerDied","Data":"db423c8cea2304d52764ded7c47b1b9f27c5a99bcb9f3667a870527f70257872"} Nov 03 17:15:22.406848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.405617 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf" Nov 03 17:15:22.420317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.420253 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerStarted","Data":"1ed22c168dfd14276e4e365b020a1ebabf1bb698b6d0c98f7352cba83612821a"} Nov 03 17:15:22.421866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.421840 6001 scope.go:122] "RemoveContainer" containerID="af6953e13084a43a21d1dd79ac958beba64436423c0bdf59fccb20bfcf205d8a" Nov 03 17:15:22.456874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.452838 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:22.529909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.529873 6001 scope.go:122] "RemoveContainer" containerID="b21117884750e6078e925f2b832830e712a9a01e4d11d507d69194fa0bf3b9fd" Nov 03 17:15:22.570567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.569438 6001 scope.go:122] "RemoveContainer" containerID="694049b2d1ec8a74749a8596ed3774a0effa650f1199332fa21ed486ba96c5d6" Nov 03 17:15:22.583404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.583323 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:22.602734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.602710 6001 scope.go:122] "RemoveContainer" containerID="69d756b61c069ed4aad1de5b025fc38548fef0fe3008cc03967a0dd9dfc0638c" Nov 03 17:15:22.619688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.619660 6001 scope.go:122] "RemoveContainer" containerID="d842d0a06b1c745615af7bb2ec2da7400d1d29189972aaf9c625beb4ad971180" Nov 03 17:15:22.633535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.633477 6001 scope.go:122] "RemoveContainer" containerID="3fa145c782111939a2acc0b7b9740e533378976e5c427023f38935b8b443ae97" Nov 03 17:15:22.664653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.664460 6001 scope.go:122] "RemoveContainer" containerID="ea32117ac8ec96e56eb633c8b119c5544e0fc8e917f970e7b55bc3cbfa1b7d97" Nov 03 17:15:22.679591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.676382 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\") pod \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " Nov 03 17:15:22.679591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.676679 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\" (UniqueName: \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\") pod \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\" (UID: \"bb33eb94-227c-4d5c-bf79-c6692cd369a1\") " Nov 03 17:15:22.681837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.681794 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca" pod "bb33eb94-227c-4d5c-bf79-c6692cd369a1" (UID: "bb33eb94-227c-4d5c-bf79-c6692cd369a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.684258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.684229 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm" pod "bb33eb94-227c-4d5c-bf79-c6692cd369a1" (UID: "bb33eb94-227c-4d5c-bf79-c6692cd369a1"). InnerVolumeSpecName "kube-api-access-n86tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.688841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.688812 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:22.695182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.695158 6001 scope.go:122] "RemoveContainer" containerID="a837bcc6a5982354897bbda474ae60ab30423c0797697317d80e00eb799dde2d" Nov 03 17:15:22.709037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.709009 6001 scope.go:122] "RemoveContainer" containerID="7018144ce047713231172d94ac86483b38a7f9e8f25837c4985193fa5c5c7c57" Nov 03 17:15:22.715987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.715956 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:22.778151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.778115 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\" (UniqueName: \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\") pod \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " Nov 03 17:15:22.778571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.778552 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\") pod \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\" (UID: \"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd\") " Nov 03 17:15:22.778965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.778930 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca" pod "2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" (UID: "2ad571d0-2095-4b3a-9c41-bd4b393fdbfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:22.779221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.779201 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.779307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.779223 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86tm\" (UniqueName: \"kubernetes.io/projected/bb33eb94-227c-4d5c-bf79-c6692cd369a1-kube-api-access-n86tm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.779307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.779232 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb33eb94-227c-4d5c-bf79-c6692cd369a1-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:22.780604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.780579 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p" pod "2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" (UID: "2ad571d0-2095-4b3a-9c41-bd4b393fdbfd"). InnerVolumeSpecName "kube-api-access-l6t8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:22.800316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.800279 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bc8532f-a07c-4360-90f0-38e4acd87103" path="/var/lib/kubelet/pods/5bc8532f-a07c-4360-90f0-38e4acd87103/volumes" Nov 03 17:15:22.801068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.801044 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" path="/var/lib/kubelet/pods/5bcd7369-352d-4d61-a346-7004aee7d0ef/volumes" Nov 03 17:15:22.826844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.826803 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:22.835043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.835011 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895435-tfkxx"] Nov 03 17:15:22.880981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:22.880877 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6t8p\" (UniqueName: \"kubernetes.io/projected/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd-kube-api-access-l6t8p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:23.213718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.189746 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:15:23.274067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.264697 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:15:23.319425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.319391 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:23.338479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.338076 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895435-svpjx"] Nov 03 17:15:23.349001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.346896 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:23.360803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.360722 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895435-hk7xf"] Nov 03 17:15:23.373407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.373369 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:23.384500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.384470 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895435-7gtwc"] Nov 03 17:15:23.390512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.390439 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:23.398986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.398952 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895435-7q2mm"] Nov 03 17:15:23.409476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.409405 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:23.415390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.415359 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1f83c0-29895435-m9n2c"] Nov 03 17:15:23.428088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.428061 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:23.442289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.442249 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" event={"ID":"bb33eb94-227c-4d5c-bf79-c6692cd369a1","Type":"ContainerDied","Data":"b1475863667d1af60521c987a5c51eba65ba54e9756f39d6421842dbecd87739"} Nov 03 17:15:23.442550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.442318 6001 scope.go:122] "RemoveContainer" containerID="4bb82d012f12b442f6c49ce94ab0ef1ac4cd22abd4567d6fca917825e72b6f59" Nov 03 17:15:23.442550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.442426 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn" Nov 03 17:15:23.459334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.458524 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" Nov 03 17:15:23.459334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.458518 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755" event={"ID":"2ad571d0-2095-4b3a-9c41-bd4b393fdbfd","Type":"ContainerDied","Data":"a8bf861b9cb98d00456f0bcd7b8118facfb2e69fc4e6bd05ede8e42db2636682"} Nov 03 17:15:23.462128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.462090 6001 generic.go:356] "Generic (PLEG): container finished" podID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerID="74378423ba7a7bfef6581919276a9c2cc68030773cddab651ab1f03d0aa39871" exitCode=60 Nov 03 17:15:23.462289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.462184 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerDied","Data":"74378423ba7a7bfef6581919276a9c2cc68030773cddab651ab1f03d0aa39871"} Nov 03 17:15:23.462695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.462670 6001 scope.go:122] "RemoveContainer" containerID="74378423ba7a7bfef6581919276a9c2cc68030773cddab651ab1f03d0aa39871" Nov 03 17:15:23.463166 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:23.463133 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q_odh-ai-gateway-infra(9a601ed0-5b71-42a3-8974-5d7c0770ffb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" Nov 03 17:15:23.471745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.471726 6001 scope.go:122] "RemoveContainer" containerID="0b34158a354821c550f22490db917e999415e51d50c71b264e636905ba0dd6ab" Nov 03 17:15:23.487288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.487253 6001 generic.go:356] "Generic (PLEG): container finished" podID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerID="1ed22c168dfd14276e4e365b020a1ebabf1bb698b6d0c98f7352cba83612821a" exitCode=60 Nov 03 17:15:23.487752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.487723 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerDied","Data":"1ed22c168dfd14276e4e365b020a1ebabf1bb698b6d0c98f7352cba83612821a"} Nov 03 17:15:23.488571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.488544 6001 scope.go:122] "RemoveContainer" containerID="1ed22c168dfd14276e4e365b020a1ebabf1bb698b6d0c98f7352cba83612821a" Nov 03 17:15:23.489255 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:23.489227 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6_odh-ai-gateway-infra(510a81ce-d3b9-48f1-9a64-0912f32376df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" Nov 03 17:15:23.506094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.504882 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerDied","Data":"6e00d7d2d0b01a9f1df3fe69ddae74cec8082e7b53f49397bb45fa4278fe9b8d"} Nov 03 17:15:23.506094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.505485 6001 scope.go:122] "RemoveContainer" containerID="6e00d7d2d0b01a9f1df3fe69ddae74cec8082e7b53f49397bb45fa4278fe9b8d" Nov 03 17:15:23.506094 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:23.506011 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx_odh-ai-gateway-infra(1c2d8a27-b230-4a8b-ab7e-498e610069ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" Nov 03 17:15:23.506499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.506475 6001 generic.go:356] "Generic (PLEG): container finished" podID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerID="6e00d7d2d0b01a9f1df3fe69ddae74cec8082e7b53f49397bb45fa4278fe9b8d" exitCode=60 Nov 03 17:15:23.514503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.514121 6001 scope.go:122] "RemoveContainer" containerID="a9ef8cd199038700d6be854a285715ddbcbaebb10053b0a583ae2899b4383861" Nov 03 17:15:23.592026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.588293 6001 scope.go:122] "RemoveContainer" containerID="71d99cd4f80951c22bd20a544e6a550c03dc82cf97776c8590e45be68fd78cd3" Nov 03 17:15:23.618696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.618634 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:23.649107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.649081 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:23.668905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.668859 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895435-gxpwn"] Nov 03 17:15:23.862610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.862520 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:23.916694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:23.905118 6001 scope.go:122] "RemoveContainer" containerID="a9e713a210078a6187992f38e7cb3d862ee19bed8ddff7215087a703f9c6eb0e" Nov 03 17:15:24.775295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:24.775254 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895435-st755"] Nov 03 17:15:25.113555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.090215 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:15:25.476880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.476830 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:25.477373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.477148 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:25.651142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.650904 6001 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:25.726115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.725626 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gf9p8" podUID="e52d3d63-9c13-44bd-beca-7370089217c4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:25.727321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:25.727040 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-sjb24" podUID="43be8571-61d3-47ea-aeb5-67d131720b19" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Nov 03 17:15:26.008620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.008574 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-njjrb" podUID="28008665-4e16-49dd-acc6-3e312861194d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:26.188496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.124940 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:26.244050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.244007 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Nov 03 17:15:26.333113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.246502 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded" Nov 03 17:15:26.522812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.393649 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:26.523932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.394227 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:26.524703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.394269 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:26.524980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.500421 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:15:26.535712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.533830 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" path="/var/lib/kubelet/pods/1e9b5cfb-83ef-409a-9ad5-641495c9a860/volumes" Nov 03 17:15:26.574231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.574102 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" path="/var/lib/kubelet/pods/2ad571d0-2095-4b3a-9c41-bd4b393fdbfd/volumes" Nov 03 17:15:26.592972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.592873 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" path="/var/lib/kubelet/pods/3fcb5379-f832-449f-827e-acce7c4c3285/volumes" Nov 03 17:15:26.594679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.594650 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" path="/var/lib/kubelet/pods/6a4eae88-7286-4d3f-9032-9984cf11eefa/volumes" Nov 03 17:15:26.601873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.601845 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" path="/var/lib/kubelet/pods/90629a0b-74c3-477e-9468-72681dd5b78d/volumes" Nov 03 17:15:26.611265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.611240 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" path="/var/lib/kubelet/pods/a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7/volumes" Nov 03 17:15:26.615863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.615831 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:26.644652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.644593 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" path="/var/lib/kubelet/pods/b7c64aac-55d0-4f28-8827-fa11441e5481/volumes" Nov 03 17:15:26.645967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.645940 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" path="/var/lib/kubelet/pods/bb33eb94-227c-4d5c-bf79-c6692cd369a1/volumes" Nov 03 17:15:26.646801 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:26.646776 6001 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.676s" Nov 03 17:15:26.651588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.651559 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:15:26.654475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.652022 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:15:26.754348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.754319 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:26.761363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.752490 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\" (UniqueName: \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\") pod \"9325a012-946b-4027-b2b6-dbb03b14b915\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " Nov 03 17:15:26.861830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.861726 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\") pod \"9325a012-946b-4027-b2b6-dbb03b14b915\" (UID: \"9325a012-946b-4027-b2b6-dbb03b14b915\") " Nov 03 17:15:26.888141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.888102 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\" (UniqueName: \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\") pod \"dbe06000-58db-4eeb-86ff-1e604357086c\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " Nov 03 17:15:26.888355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.888335 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\") pod \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " Nov 03 17:15:26.913469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.888641 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\") pod \"dbe06000-58db-4eeb-86ff-1e604357086c\" (UID: \"dbe06000-58db-4eeb-86ff-1e604357086c\") " Nov 03 17:15:26.913876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.913842 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\" (UniqueName: \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\") pod \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\" (UID: \"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef\") " Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.879093 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca" pod "9325a012-946b-4027-b2b6-dbb03b14b915" (UID: "9325a012-946b-4027-b2b6-dbb03b14b915"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.893326 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr" pod "dbe06000-58db-4eeb-86ff-1e604357086c" (UID: "dbe06000-58db-4eeb-86ff-1e604357086c"). InnerVolumeSpecName "kube-api-access-gxszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.893722 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca" pod "4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" (UID: "4ee7480b-8265-4acb-b9e3-b61dfb2b6cef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.894029 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca" pod "dbe06000-58db-4eeb-86ff-1e604357086c" (UID: "dbe06000-58db-4eeb-86ff-1e604357086c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.948077 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:26.955721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:26.948296 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.051399 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9325a012-946b-4027-b2b6-dbb03b14b915-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.051462 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxszr\" (UniqueName: \"kubernetes.io/projected/dbe06000-58db-4eeb-86ff-1e604357086c-kube-api-access-gxszr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.051481 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.051497 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbe06000-58db-4eeb-86ff-1e604357086c-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.052942 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6" pod "9325a012-946b-4027-b2b6-dbb03b14b915" (UID: "9325a012-946b-4027-b2b6-dbb03b14b915"). InnerVolumeSpecName "kube-api-access-ktnp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.053502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.053416 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97" pod "4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" (UID: "4ee7480b-8265-4acb-b9e3-b61dfb2b6cef"). InnerVolumeSpecName "kube-api-access-xzq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.109242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.109202 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:27.139075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.138970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7" event={"ID":"dbe06000-58db-4eeb-86ff-1e604357086c","Type":"ContainerDied","Data":"cdcbc726abb35fc598527f56ed37aa496386a2627dec5f70ce4497ff545ab913"} Nov 03 17:15:27.139347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.139081 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:15:27.139347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.139115 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:27.139347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.139191 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:15:27.139347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.139230 6001 scope.go:122] "RemoveContainer" containerID="4fe618d78ff21809bd6b5a0f3ff1cf8710d4802ae350cb759c34f54d7ed2e4ba" Nov 03 17:15:27.153633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.151593 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:27.173045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.170121 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzq97\" (UniqueName: \"kubernetes.io/projected/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef-kube-api-access-xzq97\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.173045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.170154 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktnp6\" (UniqueName: \"kubernetes.io/projected/9325a012-946b-4027-b2b6-dbb03b14b915-kube-api-access-ktnp6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.221672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.190807 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:27.272886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.272169 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\" (UniqueName: \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\") pod \"b91179a4-e019-4998-b071-736a3c0f4ab7\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " Nov 03 17:15:27.272886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.272438 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\" (UniqueName: \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\") pod \"b9b61654-878c-4b35-9b1f-e25bb66698ce\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " Nov 03 17:15:27.272886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.272549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\") pod \"b9b61654-878c-4b35-9b1f-e25bb66698ce\" (UID: \"b9b61654-878c-4b35-9b1f-e25bb66698ce\") " Nov 03 17:15:27.272886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.272645 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\") pod \"b91179a4-e019-4998-b071-736a3c0f4ab7\" (UID: \"b91179a4-e019-4998-b071-736a3c0f4ab7\") " Nov 03 17:15:27.284314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.284054 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca" pod "b9b61654-878c-4b35-9b1f-e25bb66698ce" (UID: "b9b61654-878c-4b35-9b1f-e25bb66698ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.293928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.293892 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca" pod "b91179a4-e019-4998-b071-736a3c0f4ab7" (UID: "b91179a4-e019-4998-b071-736a3c0f4ab7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.366285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.364931 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf" pod "b91179a4-e019-4998-b071-736a3c0f4ab7" (UID: "b91179a4-e019-4998-b071-736a3c0f4ab7"). InnerVolumeSpecName "kube-api-access-tb6bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.366285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.365038 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6" pod "b9b61654-878c-4b35-9b1f-e25bb66698ce" (UID: "b9b61654-878c-4b35-9b1f-e25bb66698ce"). InnerVolumeSpecName "kube-api-access-xpmh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.384625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.384586 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb6bf\" (UniqueName: \"kubernetes.io/projected/b91179a4-e019-4998-b071-736a3c0f4ab7-kube-api-access-tb6bf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.384860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.384843 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpmh6\" (UniqueName: \"kubernetes.io/projected/b9b61654-878c-4b35-9b1f-e25bb66698ce-kube-api-access-xpmh6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.384962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.384950 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b61654-878c-4b35-9b1f-e25bb66698ce-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.385058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.385046 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91179a4-e019-4998-b071-736a3c0f4ab7-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.569948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.569910 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:27.667272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.661488 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:27.669953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.669858 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895435-w9jf7"] Nov 03 17:15:27.682217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.682193 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:27.692942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.692910 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\" (UniqueName: \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\") pod \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " Nov 03 17:15:27.693253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.693230 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\") pod \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\" (UID: \"1c2d8a27-b230-4a8b-ab7e-498e610069ad\") " Nov 03 17:15:27.695715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.695684 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca" pod "1c2d8a27-b230-4a8b-ab7e-498e610069ad" (UID: "1c2d8a27-b230-4a8b-ab7e-498e610069ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.696734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.696708 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:27.698821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.698801 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:27.702550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.701484 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs" pod "1c2d8a27-b230-4a8b-ab7e-498e610069ad" (UID: "1c2d8a27-b230-4a8b-ab7e-498e610069ad"). InnerVolumeSpecName "kube-api-access-mnxcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.795024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.794992 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\" (UniqueName: \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\") pod \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " Nov 03 17:15:27.795231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795041 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\") pod \"510a81ce-d3b9-48f1-9a64-0912f32376df\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " Nov 03 17:15:27.795231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795072 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\" (UniqueName: \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\") pod \"510a81ce-d3b9-48f1-9a64-0912f32376df\" (UID: \"510a81ce-d3b9-48f1-9a64-0912f32376df\") " Nov 03 17:15:27.795231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795196 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\") pod \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\" (UID: \"9a601ed0-5b71-42a3-8974-5d7c0770ffb1\") " Nov 03 17:15:27.795497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795238 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\" (UniqueName: \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\") pod \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " Nov 03 17:15:27.795497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795321 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\") pod \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\" (UID: \"9a14583c-2fca-4c77-b73b-8a84f18e2c43\") " Nov 03 17:15:27.795818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795797 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c2d8a27-b230-4a8b-ab7e-498e610069ad-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.795967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.795948 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxcs\" (UniqueName: \"kubernetes.io/projected/1c2d8a27-b230-4a8b-ab7e-498e610069ad-kube-api-access-mnxcs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.796268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.796234 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca" pod "9a601ed0-5b71-42a3-8974-5d7c0770ffb1" (UID: "9a601ed0-5b71-42a3-8974-5d7c0770ffb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.796356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.796267 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca" pod "9a14583c-2fca-4c77-b73b-8a84f18e2c43" (UID: "9a14583c-2fca-4c77-b73b-8a84f18e2c43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.796717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.796690 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca" pod "510a81ce-d3b9-48f1-9a64-0912f32376df" (UID: "510a81ce-d3b9-48f1-9a64-0912f32376df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:27.798518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.798496 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc" pod "510a81ce-d3b9-48f1-9a64-0912f32376df" (UID: "510a81ce-d3b9-48f1-9a64-0912f32376df"). InnerVolumeSpecName "kube-api-access-dxczc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.798601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.798528 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9" pod "9a601ed0-5b71-42a3-8974-5d7c0770ffb1" (UID: "9a601ed0-5b71-42a3-8974-5d7c0770ffb1"). InnerVolumeSpecName "kube-api-access-24sk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.798733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.798720 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6" pod "9a14583c-2fca-4c77-b73b-8a84f18e2c43" (UID: "9a14583c-2fca-4c77-b73b-8a84f18e2c43"). InnerVolumeSpecName "kube-api-access-zmwf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:27.899616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899535 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.899616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899573 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwf6\" (UniqueName: \"kubernetes.io/projected/9a14583c-2fca-4c77-b73b-8a84f18e2c43-kube-api-access-zmwf6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.899616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899590 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a14583c-2fca-4c77-b73b-8a84f18e2c43-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.899616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899606 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24sk9\" (UniqueName: \"kubernetes.io/projected/9a601ed0-5b71-42a3-8974-5d7c0770ffb1-kube-api-access-24sk9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.899616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899623 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510a81ce-d3b9-48f1-9a64-0912f32376df-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:27.900106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:27.899638 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxczc\" (UniqueName: \"kubernetes.io/projected/510a81ce-d3b9-48f1-9a64-0912f32376df-kube-api-access-dxczc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:15:28.129032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.128939 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" Nov 03 17:15:28.129191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.128939 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9" event={"ID":"b91179a4-e019-4998-b071-736a3c0f4ab7","Type":"ContainerDied","Data":"563867ec7f35c61f343a4d9ab60b88d61acc0c77d2cad3c1b890bb6153d87671"} Nov 03 17:15:28.129191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.129103 6001 scope.go:122] "RemoveContainer" containerID="c456f9e607cb434c15b2585caa3cc0d6e08eb05c450da96579124b5844e216da" Nov 03 17:15:28.130934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.130905 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" event={"ID":"9325a012-946b-4027-b2b6-dbb03b14b915","Type":"ContainerDied","Data":"809510a94e7c0bf0e46e908e1c6c8a8036b035f0eaab60a8432afb5d465acb05"} Nov 03 17:15:28.131085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.131067 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6" Nov 03 17:15:28.133075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.133049 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" event={"ID":"1c2d8a27-b230-4a8b-ab7e-498e610069ad","Type":"ContainerDied","Data":"7db4394639362967f2b7daece03d79c61441ca6ac5747a7b989eaacd5460a720"} Nov 03 17:15:28.133204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.133190 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx" Nov 03 17:15:28.140499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.138329 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" Nov 03 17:15:28.140499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.138332 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29" event={"ID":"b9b61654-878c-4b35-9b1f-e25bb66698ce","Type":"ContainerDied","Data":"47121ab85371340ea45f9bea5d3087a6c545c233a9b1abb5da1638f7ae2acd3e"} Nov 03 17:15:28.141028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.140514 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" event={"ID":"4ee7480b-8265-4acb-b9e3-b61dfb2b6cef","Type":"ContainerDied","Data":"596ea314d296c1556e2875aef3134e6d52f53339c8256802383699bbf1122503"} Nov 03 17:15:28.141028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.140672 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7" Nov 03 17:15:28.144332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.144304 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" event={"ID":"9a601ed0-5b71-42a3-8974-5d7c0770ffb1","Type":"ContainerDied","Data":"741aed863538ac0c6458bee2b96113c343ef927135281587a9b4202055cd7d71"} Nov 03 17:15:28.144632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.144389 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q" Nov 03 17:15:28.148576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.148418 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" event={"ID":"9a14583c-2fca-4c77-b73b-8a84f18e2c43","Type":"ContainerDied","Data":"79dd8a201263f881b879c2c4b5ea10bc7754d5bd5b359adef1034bb2b2bc6dba"} Nov 03 17:15:28.148576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.148531 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h" Nov 03 17:15:28.149413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.149277 6001 scope.go:122] "RemoveContainer" containerID="c57489903676cf372388f194b01ecfc054fa6aeb7deb24d33b78131c2bb3d0ae" Nov 03 17:15:28.153227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.153206 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" Nov 03 17:15:28.153965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.153907 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6" event={"ID":"510a81ce-d3b9-48f1-9a64-0912f32376df","Type":"ContainerDied","Data":"dd46831d68c81beaceb474c397da1f6fef50157c39ed2288e16c05cc146b3bed"} Nov 03 17:15:28.167106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.167077 6001 scope.go:122] "RemoveContainer" containerID="6e00d7d2d0b01a9f1df3fe69ddae74cec8082e7b53f49397bb45fa4278fe9b8d" Nov 03 17:15:28.183856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.183834 6001 scope.go:122] "RemoveContainer" containerID="fc77c419ed40debcef3e7bcfbff702313a964d79ad80c07b0851dc5b3dd31c74" Nov 03 17:15:28.203252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.203225 6001 scope.go:122] "RemoveContainer" containerID="a47a0789fb245f62befe90613e37fc5739246e24b8af5a2a4d0919a3f42454b4" Nov 03 17:15:28.217704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.217680 6001 scope.go:122] "RemoveContainer" containerID="74378423ba7a7bfef6581919276a9c2cc68030773cddab651ab1f03d0aa39871" Nov 03 17:15:28.231067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.231032 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:28.233896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.233877 6001 scope.go:122] "RemoveContainer" containerID="3ca0b2b29444613b67917380f8eb7aedabcfaed9d1b0b1847a3078d4ccdefcb9" Nov 03 17:15:28.237614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.237590 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895435-vvjm6"] Nov 03 17:15:28.247280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.247260 6001 scope.go:122] "RemoveContainer" containerID="1ed22c168dfd14276e4e365b020a1ebabf1bb698b6d0c98f7352cba83612821a" Nov 03 17:15:28.322110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.322048 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:28.328480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.328431 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895435-f8d6h"] Nov 03 17:15:28.363809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.363772 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:28.368892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.368863 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895435-xx2bx"] Nov 03 17:15:28.402171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.402094 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:28.407028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.406998 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895435-gdq9q"] Nov 03 17:15:28.445985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.445955 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:28.450941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.450915 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895435-m6l29"] Nov 03 17:15:28.492332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.491767 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:28.497760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.497734 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895435-b4tk9"] Nov 03 17:15:28.536130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.536101 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:28.541705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.541677 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895435-r2bb7"] Nov 03 17:15:28.582204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.582170 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:28.591281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.591255 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895435-tqtq6"] Nov 03 17:15:28.796205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.796160 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" path="/var/lib/kubelet/pods/1c2d8a27-b230-4a8b-ab7e-498e610069ad/volumes" Nov 03 17:15:28.796837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.796812 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" path="/var/lib/kubelet/pods/4ee7480b-8265-4acb-b9e3-b61dfb2b6cef/volumes" Nov 03 17:15:28.797355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.797334 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" path="/var/lib/kubelet/pods/510a81ce-d3b9-48f1-9a64-0912f32376df/volumes" Nov 03 17:15:28.798226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.798202 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" path="/var/lib/kubelet/pods/9325a012-946b-4027-b2b6-dbb03b14b915/volumes" Nov 03 17:15:28.798762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.798741 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" path="/var/lib/kubelet/pods/9a14583c-2fca-4c77-b73b-8a84f18e2c43/volumes" Nov 03 17:15:28.799288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.799265 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" path="/var/lib/kubelet/pods/9a601ed0-5b71-42a3-8974-5d7c0770ffb1/volumes" Nov 03 17:15:28.800057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.800034 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" path="/var/lib/kubelet/pods/b91179a4-e019-4998-b071-736a3c0f4ab7/volumes" Nov 03 17:15:28.800606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.800586 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" path="/var/lib/kubelet/pods/b9b61654-878c-4b35-9b1f-e25bb66698ce/volumes" Nov 03 17:15:28.801125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:28.801104 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" path="/var/lib/kubelet/pods/dbe06000-58db-4eeb-86ff-1e604357086c/volumes" Nov 03 17:15:52.289496 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:15:52.289437 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:15:54.627933 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:54.627895 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:15:54.628458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:54.628137 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" containerID="cri-o://97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42" gracePeriod=30 Nov 03 17:15:56.042127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:56.042087 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:15:56.042598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:56.042309 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" containerID="cri-o://1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791" gracePeriod=30 Nov 03 17:15:59.933829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:15:59.933800 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:16:00.072362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072324 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072395 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072463 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072519 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072576 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072622 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072689 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072851 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.072964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.072934 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\") pod \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\" (UID: \"1a8aa919-a43e-44b1-a9cc-d884d990a7ae\") " Nov 03 17:16:00.074387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.074030 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:00.077638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.076587 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:00.077638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.076782 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:00.077638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.077109 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:16:00.077638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.077589 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:00.092721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.091563 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:16:00.102326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.102288 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "kube-api-access-bqkh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:00.102550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.102351 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:00.105092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.105055 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy" pod "1a8aa919-a43e-44b1-a9cc-d884d990a7ae" (UID: "1a8aa919-a43e-44b1-a9cc-d884d990a7ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176020 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176058 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176070 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176084 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176093 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqkh2\" (UniqueName: \"kubernetes.io/projected/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-kube-api-access-bqkh2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176102 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176110 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176120 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.176559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.176128 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8aa919-a43e-44b1-a9cc-d884d990a7ae-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:00.693868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.693836 6001 generic.go:356] "Generic (PLEG): container finished" podID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerID="97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42" exitCode=0 Nov 03 17:16:00.694059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.693888 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" Nov 03 17:16:00.694059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.693919 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" event={"ID":"1a8aa919-a43e-44b1-a9cc-d884d990a7ae","Type":"ContainerDied","Data":"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42"} Nov 03 17:16:00.694059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.693975 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw" event={"ID":"1a8aa919-a43e-44b1-a9cc-d884d990a7ae","Type":"ContainerDied","Data":"8ac55c7cc6d8397fca7ae5e538ea6304f4af0cd11aaf88cd2a780b06ca769264"} Nov 03 17:16:00.694059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.694009 6001 scope.go:122] "RemoveContainer" containerID="97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42" Nov 03 17:16:00.708172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.708155 6001 scope.go:122] "RemoveContainer" containerID="97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42" Nov 03 17:16:00.708477 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:00.708439 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42\": container with ID starting with 97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42 not found: ID does not exist" containerID="97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42" Nov 03 17:16:00.708555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.708485 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42"} err="failed to get container status \"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42\": rpc error: code = NotFound desc = could not find container \"97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42\": container with ID starting with 97a09c93e010b32952da718110e32b1501e4ca1d46934f48afca8d7b0f2c1d42 not found: ID does not exist" Nov 03 17:16:00.776663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.776483 6001 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:16:00.790638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.789275 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:16:00.801035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.801005 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:16:00.807394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:00.807355 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-r9qnw"] Nov 03 17:16:01.362216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.362189 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:16:01.403267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403241 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403274 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403310 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403328 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403415 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403485 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403571 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:01.403618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403587 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:01.403854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403688 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403737 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.403854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403816 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:01.403854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403833 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\") pod \"158091eb-b54f-4d18-8388-1904b664d3c1\" (UID: \"158091eb-b54f-4d18-8388-1904b664d3c1\") " Nov 03 17:16:01.404079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.403902 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:16:01.404079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404028 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:01.404912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404885 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.404912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404914 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.404912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404928 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/158091eb-b54f-4d18-8388-1904b664d3c1-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.405204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404941 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.405204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.404957 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.405800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.405775 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:01.405896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.405879 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "kube-api-access-rgcmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:01.406052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.406029 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:16:01.406124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.406059 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token" pod "158091eb-b54f-4d18-8388-1904b664d3c1" (UID: "158091eb-b54f-4d18-8388-1904b664d3c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:01.507197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.507161 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgcmd\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-kube-api-access-rgcmd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.507197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.507192 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/158091eb-b54f-4d18-8388-1904b664d3c1-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.507197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.507203 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/158091eb-b54f-4d18-8388-1904b664d3c1-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.507438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.507211 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/158091eb-b54f-4d18-8388-1904b664d3c1-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:01.720809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.720704 6001 generic.go:356] "Generic (PLEG): container finished" podID="158091eb-b54f-4d18-8388-1904b664d3c1" containerID="1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791" exitCode=0 Nov 03 17:16:01.720809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.720748 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" event={"ID":"158091eb-b54f-4d18-8388-1904b664d3c1","Type":"ContainerDied","Data":"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791"} Nov 03 17:16:01.720809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.720778 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" Nov 03 17:16:01.721077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.720853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts" event={"ID":"158091eb-b54f-4d18-8388-1904b664d3c1","Type":"ContainerDied","Data":"09f44e351de2262f5310d776e98d39c19d1ebb46a882a5170a0253d39c21fb63"} Nov 03 17:16:01.721077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.720881 6001 scope.go:122] "RemoveContainer" containerID="1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791" Nov 03 17:16:01.737428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.737406 6001 scope.go:122] "RemoveContainer" containerID="1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791" Nov 03 17:16:01.737783 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:01.737763 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791\": container with ID starting with 1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791 not found: ID does not exist" containerID="1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791" Nov 03 17:16:01.737839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.737792 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791"} err="failed to get container status \"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791\": rpc error: code = NotFound desc = could not find container \"1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791\": container with ID starting with 1b5e139b460ce846b14aa615e3297a46da5a0024a06331d361aa6034a1a5e791 not found: ID does not exist" Nov 03 17:16:01.781067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.781022 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:16:01.786507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:01.786475 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-9f6ts"] Nov 03 17:16:02.792949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:02.792913 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" path="/var/lib/kubelet/pods/158091eb-b54f-4d18-8388-1904b664d3c1/volumes" Nov 03 17:16:02.793554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:02.793537 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" path="/var/lib/kubelet/pods/1a8aa919-a43e-44b1-a9cc-d884d990a7ae/volumes" Nov 03 17:16:11.857547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.857511 6001 scope.go:122] "RemoveContainer" containerID="2be167a3a413ff7324155019ff83131d8ec2ca4facdd6d18a7483fa854d4bde9" Nov 03 17:16:11.870247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.870212 6001 scope.go:122] "RemoveContainer" containerID="2ce75fefcef8d13ead4385987906165dccbebb07ebbd08ecf2017fc79c9b2a1e" Nov 03 17:16:11.885935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.885910 6001 scope.go:122] "RemoveContainer" containerID="b41c76b0a65ce8d7bc0478e4f9448210a48fcf93b3b3397682e3f41300486f16" Nov 03 17:16:11.900143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.900114 6001 scope.go:122] "RemoveContainer" containerID="dae35493b4b8eddf3550a427c447862193c7619476689708bb27f61cd3064e59" Nov 03 17:16:11.911863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.911839 6001 scope.go:122] "RemoveContainer" containerID="9144ce277d50b7b66c9b8a0552bebe351f02e0d3a37e09d93d7da963adf14fd6" Nov 03 17:16:11.923686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.923662 6001 scope.go:122] "RemoveContainer" containerID="3a3e48d15bbf57a77700ae5a65f9d7ace99276b33d8c8893f3b645069827707f" Nov 03 17:16:11.937665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:11.937636 6001 scope.go:122] "RemoveContainer" containerID="489af7573ecb3bbaf887919fd633b8663ed75a3fde0d54914c2d29c524b2a520" Nov 03 17:16:21.039464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.039410 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:16:21.040568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040539 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.040568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040565 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040578 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040584 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040593 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040599 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040609 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040616 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040626 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040634 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040645 6001 cpu_manager.go:414] "containerMap: removing container" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040650 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040656 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040660 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040669 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040673 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040681 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040686 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040696 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040701 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040709 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040714 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040720 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040724 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040729 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040733 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040738 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040743 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040756 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040760 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040770 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040776 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040782 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040786 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040794 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040798 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040806 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040810 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040816 6001 cpu_manager.go:414] "containerMap: removing container" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040821 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040829 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040834 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.040814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040844 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040851 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040864 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040870 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040880 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040885 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040892 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040896 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040907 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040911 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040918 6001 cpu_manager.go:414] "containerMap: removing container" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040924 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040933 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040938 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040951 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040955 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040961 6001 cpu_manager.go:414] "containerMap: removing container" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040965 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040972 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040976 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040982 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040986 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040993 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.040998 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041006 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041011 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041023 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041028 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041035 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041040 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041049 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041053 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041065 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041069 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041076 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041080 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041086 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041090 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041104 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041108 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041122 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041126 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041133 6001 cpu_manager.go:414] "containerMap: removing container" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041138 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041144 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041149 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041160 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041164 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041171 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041175 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041326 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041334 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041341 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041348 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041353 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041358 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041363 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041370 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041375 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041383 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041389 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041394 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041400 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041405 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041411 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041416 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041423 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041429 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041436 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041456 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041463 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041469 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041475 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="158091eb-b54f-4d18-8388-1904b664d3c1" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041480 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041485 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041491 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041499 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041505 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041512 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041519 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041524 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041530 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041536 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041543 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041548 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041552 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041558 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4eae88-7286-4d3f-9032-9984cf11eefa" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041566 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041571 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9325a012-946b-4027-b2b6-dbb03b14b915" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041577 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14583c-2fca-4c77-b73b-8a84f18e2c43" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041584 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041595 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041602 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041609 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8aa919-a43e-44b1-a9cc-d884d990a7ae" containerName="istio-proxy" Nov 03 17:16:21.043480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.041615 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fcb5379-f832-449f-827e-acce7c4c3285" containerName="cleanup" Nov 03 17:16:21.050010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.049976 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.083465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.080098 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-527051-openshift-default-dockercfg-wcccz\"" Nov 03 17:16:21.083465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.083398 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:16:21.135295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135253 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.135532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135358 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.135691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135531 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.135691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135632 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.135820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135693 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.135820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.135751 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.136033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.136005 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.136089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.136076 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.136212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.136194 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh44b\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.251707 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.251816 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252011 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252068 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252146 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh44b\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252205 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252261 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252348 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.252472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.252437 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.275469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.253343 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.275469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.263046 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.275469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.264121 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.275469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.264396 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.275469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.264682 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.283478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.282314 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.307852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.306667 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.337470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.330250 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh44b\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.369467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.366094 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\") pod \"e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.382810 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:21.832343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.832303 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:16:21.835589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.835556 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:21.835693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.835623 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:21.835693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:21.835654 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:22.067818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.067780 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerStarted","Data":"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697"} Nov 03 17:16:22.068263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.067847 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerStarted","Data":"c4205fa6fe670d68345325fd327bba99fbe462020f93740c45700cc918e25f9e"} Nov 03 17:16:22.129103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.128760 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podStartSLOduration=2.128740638 podStartE2EDuration="2.128740638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:22.122930034 +0000 UTC m=+3677.456143606" watchObservedRunningTime="2026-11-03 17:16:22.128740638 +0000 UTC m=+3677.461954208" Nov 03 17:16:22.335492 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:22.335416 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:16:22.383866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.383763 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:22.385713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.385681 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:16:22.385901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:22.385740 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:16:23.389847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:23.389814 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:24.088709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:24.088680 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:24.089523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:24.089499 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:16:26.818009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.817969 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:16:26.818981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.818958 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:26.818981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.818979 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819004 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819009 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819020 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819025 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819033 6001 cpu_manager.go:414] "containerMap: removing container" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819038 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819051 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819056 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819069 6001 cpu_manager.go:414] "containerMap: removing container" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819074 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819096 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819101 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb33eb94-227c-4d5c-bf79-c6692cd369a1" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819109 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819112 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819139 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819143 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bcd7369-352d-4d61-a346-7004aee7d0ef" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819154 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:26.819188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819158 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b7b2e1-019e-4b2c-bfa5-4ef2397718c7" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819280 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ad571d0-2095-4b3a-9c41-bd4b393fdbfd" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819293 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91179a4-e019-4998-b071-736a3c0f4ab7" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819301 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c2d8a27-b230-4a8b-ab7e-498e610069ad" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819309 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b61654-878c-4b35-9b1f-e25bb66698ce" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819316 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="90629a0b-74c3-477e-9468-72681dd5b78d" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819326 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a601ed0-5b71-42a3-8974-5d7c0770ffb1" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819332 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="510a81ce-d3b9-48f1-9a64-0912f32376df" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819340 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbe06000-58db-4eeb-86ff-1e604357086c" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819347 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee7480b-8265-4acb-b9e3-b61dfb2b6cef" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819354 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7c64aac-55d0-4f28-8827-fa11441e5481" containerName="cleanup" Nov 03 17:16:26.819905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.819365 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9b5cfb-83ef-409a-9ad5-641495c9a860" containerName="cleanup" Nov 03 17:16:26.823783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.823753 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:26.845788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.845757 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-527051-dockercfg-lmhj8\"" Nov 03 17:16:26.847495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.847468 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:16:26.859672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.856617 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-527051\"" Nov 03 17:16:26.875236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.875189 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5rf9\" (UniqueName: \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:26.875647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.875616 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:26.916987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.916952 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:16:26.922201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.922174 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:26.947387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:26.947342 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:16:27.039594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.027538 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5rf9\" (UniqueName: \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:27.039594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.027902 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:27.039594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.027953 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrgb\" (UniqueName: \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.039594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.028024 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.195842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.152006 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrgb\" (UniqueName: \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.195842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.152106 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.195842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.153488 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.195842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.193001 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5rf9\" (UniqueName: \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:27.195842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.193070 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:27.236101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.236025 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrgb\" (UniqueName: \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\") pod \"payload-processing-e2e-inf-b-527051-788c8b4495-zccjn\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.270516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.259613 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:27.500323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.500290 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:27.959379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:27.959339 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l" podUID="a7aa56f7-e028-49dc-b1df-c6437a0096d6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded" Nov 03 17:16:28.296018 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:28.295988 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e754262_de1f_45d8_8d8a_dab2b7985a22.slice/crio-ee0c852f49ecba5d0b17f9b115c925446ddf7a1a99b04f71422b73c5682b2d41 WatchSource:0}: Error finding container ee0c852f49ecba5d0b17f9b115c925446ddf7a1a99b04f71422b73c5682b2d41: Status 404 returned error can't find the container with id ee0c852f49ecba5d0b17f9b115c925446ddf7a1a99b04f71422b73c5682b2d41 Nov 03 17:16:28.297802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.297774 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:16:28.406895 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:28.406859 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ac7f38_1526_4247_8d50_9625702dd2dd.slice/crio-34f380139b01d3eb3a574d62c364830e209bc11c49d6d7ed12451f2239a656f1 WatchSource:0}: Error finding container 34f380139b01d3eb3a574d62c364830e209bc11c49d6d7ed12451f2239a656f1: Status 404 returned error can't find the container with id 34f380139b01d3eb3a574d62c364830e209bc11c49d6d7ed12451f2239a656f1 Nov 03 17:16:28.408532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.408504 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:16:28.513964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.513933 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" event={"ID":"93ac7f38-1526-4247-8d50-9625702dd2dd","Type":"ContainerStarted","Data":"34f380139b01d3eb3a574d62c364830e209bc11c49d6d7ed12451f2239a656f1"} Nov 03 17:16:28.516055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.516029 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" event={"ID":"2e754262-de1f-45d8-8d8a-dab2b7985a22","Type":"ContainerStarted","Data":"09a407cfff16dbccd4734b028cf7cc815829a18b02acb3e5226d6c4e04c5b9a4"} Nov 03 17:16:28.516170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.516075 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" event={"ID":"2e754262-de1f-45d8-8d8a-dab2b7985a22","Type":"ContainerStarted","Data":"ee0c852f49ecba5d0b17f9b115c925446ddf7a1a99b04f71422b73c5682b2d41"} Nov 03 17:16:28.516235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.516170 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:28.581322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:28.581100 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" podStartSLOduration=2.5810820420000002 podStartE2EDuration="2.581082042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:28.576100556 +0000 UTC m=+3683.909314125" watchObservedRunningTime="2026-11-03 17:16:28.581082042 +0000 UTC m=+3683.914295614" Nov 03 17:16:29.527603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:29.527563 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" event={"ID":"93ac7f38-1526-4247-8d50-9625702dd2dd","Type":"ContainerStarted","Data":"1c4fc2e46920969418d1f3c8f29e98697a09ef9efddb854b8c996790fbb43737"} Nov 03 17:16:29.528029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:29.527626 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:29.574989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:29.574928 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" podStartSLOduration=3.574914228 podStartE2EDuration="3.574914228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:29.572248689 +0000 UTC m=+3684.905462278" watchObservedRunningTime="2026-11-03 17:16:29.574914228 +0000 UTC m=+3684.908127797" Nov 03 17:16:30.909503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.909462 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp"] Nov 03 17:16:30.915384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.915360 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:30.970673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.970637 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:30.971221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.971185 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:30.971576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.971551 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm427\" (UniqueName: \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:30.980305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.980275 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:16:30.980723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.980702 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:16:30.981001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.980981 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:16:30.981234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:30.981217 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:16:31.000819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.000761 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp"] Nov 03 17:16:31.074233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.074166 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm427\" (UniqueName: \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.074496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.074411 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.074596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.074537 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.075287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.075260 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.077003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.076979 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.102381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.102356 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm427\" (UniqueName: \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\") pod \"maas-api-revoke-sub-c365dc46-xs7bp\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.309302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.309268 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:31.623527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:31.623336 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp"] Nov 03 17:16:32.609270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.609203 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" event={"ID":"61785f8e-ad91-494a-ba1a-0744ce7ed33b","Type":"ContainerStarted","Data":"af82418287dfd0bce58c5a4143a9bb33553e06b09e43860e49d017180c1e5734"} Nov 03 17:16:32.611250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.609302 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" event={"ID":"61785f8e-ad91-494a-ba1a-0744ce7ed33b","Type":"ContainerStarted","Data":"65a378a9737ce8a940546d5594b84aae4daf1cbd521e366f1096cdd5fe98911e"} Nov 03 17:16:32.654087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.654028 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" podStartSLOduration=2.654014115 podStartE2EDuration="2.654014115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:32.652788833 +0000 UTC m=+3687.986002417" watchObservedRunningTime="2026-11-03 17:16:32.654014115 +0000 UTC m=+3687.987227683" Nov 03 17:16:32.938939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.938846 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:16:32.945871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.945839 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:32.972618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.972582 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-236539-openshift-default-dockercfg-nkccc\"" Nov 03 17:16:32.976879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:32.976847 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.132937 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133019 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g49\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133067 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133139 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133174 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133238 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133314 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133400 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.155470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.133649 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.235729 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.235807 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g49\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.235849 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.235920 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.235953 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.236017 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.236096 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.236182 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.236351 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.245466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.237323 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.257475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.252875 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.257475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.253142 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.257475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.253361 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.257475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.253952 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.273464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.272817 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.284349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.284040 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.293478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.289416 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g49\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.331223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.322946 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\") pod \"e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.569788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.569683 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:33.621521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.621486 6001 generic.go:356] "Generic (PLEG): container finished" podID="61785f8e-ad91-494a-ba1a-0744ce7ed33b" containerID="af82418287dfd0bce58c5a4143a9bb33553e06b09e43860e49d017180c1e5734" exitCode=0 Nov 03 17:16:33.621956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.621549 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" event={"ID":"61785f8e-ad91-494a-ba1a-0744ce7ed33b","Type":"ContainerDied","Data":"af82418287dfd0bce58c5a4143a9bb33553e06b09e43860e49d017180c1e5734"} Nov 03 17:16:33.901309 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:33.901279 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863f8808_b938_4728_86f4_ec5ad8c7758a.slice/crio-9bab48ca14e8e6575a37c34427f10d6bdc899a5a64295c60207b4d901524a3ad WatchSource:0}: Error finding container 9bab48ca14e8e6575a37c34427f10d6bdc899a5a64295c60207b4d901524a3ad: Status 404 returned error can't find the container with id 9bab48ca14e8e6575a37c34427f10d6bdc899a5a64295c60207b4d901524a3ad Nov 03 17:16:33.904077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.904054 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:16:33.904790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.904761 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:33.904857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.904828 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:33.904895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:33.904878 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:34.632018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.631980 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerStarted","Data":"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c"} Nov 03 17:16:34.632559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.632040 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerStarted","Data":"9bab48ca14e8e6575a37c34427f10d6bdc899a5a64295c60207b4d901524a3ad"} Nov 03 17:16:34.698292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.698236 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podStartSLOduration=2.698220712 podStartE2EDuration="2.698220712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:34.694907218 +0000 UTC m=+3690.028120787" watchObservedRunningTime="2026-11-03 17:16:34.698220712 +0000 UTC m=+3690.031434345" Nov 03 17:16:34.866774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.866266 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:34.978603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.978570 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\") pod \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " Nov 03 17:16:34.978815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.978707 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\" (UniqueName: \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\") pod \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " Nov 03 17:16:34.978815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.978812 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\") pod \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\" (UID: \"61785f8e-ad91-494a-ba1a-0744ce7ed33b\") " Nov 03 17:16:34.979285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.979256 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca" pod "61785f8e-ad91-494a-ba1a-0744ce7ed33b" (UID: "61785f8e-ad91-494a-ba1a-0744ce7ed33b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:16:34.979741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.979712 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:34.981008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.980976 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca" pod "61785f8e-ad91-494a-ba1a-0744ce7ed33b" (UID: "61785f8e-ad91-494a-ba1a-0744ce7ed33b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:16:34.981717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:34.981692 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427" pod "61785f8e-ad91-494a-ba1a-0744ce7ed33b" (UID: "61785f8e-ad91-494a-ba1a-0744ce7ed33b"). InnerVolumeSpecName "kube-api-access-zm427". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:35.081485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.081411 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm427\" (UniqueName: \"kubernetes.io/projected/61785f8e-ad91-494a-ba1a-0744ce7ed33b-kube-api-access-zm427\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:35.081485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.081455 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61785f8e-ad91-494a-ba1a-0744ce7ed33b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:35.570494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.570457 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:35.578158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.578133 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:35.643804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.643770 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" Nov 03 17:16:35.644302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.643805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp" event={"ID":"61785f8e-ad91-494a-ba1a-0744ce7ed33b","Type":"ContainerDied","Data":"65a378a9737ce8a940546d5594b84aae4daf1cbd521e366f1096cdd5fe98911e"} Nov 03 17:16:35.644302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.643855 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65a378a9737ce8a940546d5594b84aae4daf1cbd521e366f1096cdd5fe98911e" Nov 03 17:16:35.644427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.644354 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:35.645340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:35.645318 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:16:39.530837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:39.530801 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:16:40.050541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.050495 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:16:40.051964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.051942 6001 cpu_manager.go:414] "containerMap: removing container" podUID="61785f8e-ad91-494a-ba1a-0744ce7ed33b" containerName="revoke-keys" Nov 03 17:16:40.052043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.051967 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61785f8e-ad91-494a-ba1a-0744ce7ed33b" containerName="revoke-keys" Nov 03 17:16:40.052310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.052297 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="61785f8e-ad91-494a-ba1a-0744ce7ed33b" containerName="revoke-keys" Nov 03 17:16:40.057461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.057423 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.073827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.073795 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-236539-dockercfg-8lzcv\"" Nov 03 17:16:40.073992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.073857 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-236539\"" Nov 03 17:16:40.077327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.077269 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:16:40.091922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.091888 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.095790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.095750 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9m2\" (UniqueName: \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.244469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.239329 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9m2\" (UniqueName: \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.244469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.239740 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.244469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.240818 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.452536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.450708 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:16:40.489862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.489827 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:16:40.494695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.491559 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.494695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.493129 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9m2\" (UniqueName: \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\") pod \"payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.542383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.542351 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:16:40.554405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.554366 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.554618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.554436 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dpqh\" (UniqueName: \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.656232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.656192 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.656437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.656257 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dpqh\" (UniqueName: \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.657027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.657001 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.681102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.681067 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dpqh\" (UniqueName: \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\") pod \"payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:40.681947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.681922 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:40.814347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:40.814307 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:41.008637 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:41.008608 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17883199_964b_4c64_a594_246fde087c55.slice/crio-9285ac63aba881761f25d9c80190b673afac68bc62b412d6f2ffc430b9e6e5df WatchSource:0}: Error finding container 9285ac63aba881761f25d9c80190b673afac68bc62b412d6f2ffc430b9e6e5df: Status 404 returned error can't find the container with id 9285ac63aba881761f25d9c80190b673afac68bc62b412d6f2ffc430b9e6e5df Nov 03 17:16:41.010740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.010719 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:16:41.143734 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:41.143696 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7869d404_f872_4e91_974c_2ed264c1c0b7.slice/crio-a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106 WatchSource:0}: Error finding container a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106: Status 404 returned error can't find the container with id a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106 Nov 03 17:16:41.145687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.145573 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:16:41.773932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.773818 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" event={"ID":"7869d404-f872-4e91-974c-2ed264c1c0b7","Type":"ContainerStarted","Data":"8442b3ac5fee0cea320d277962183e9255bfd31c5fbfc3e876ab107e7e431c39"} Nov 03 17:16:41.773932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.773889 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" event={"ID":"7869d404-f872-4e91-974c-2ed264c1c0b7","Type":"ContainerStarted","Data":"a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106"} Nov 03 17:16:41.776340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.776301 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" event={"ID":"17883199-964b-4c64-a594-246fde087c55","Type":"ContainerStarted","Data":"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b"} Nov 03 17:16:41.776552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.776366 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" event={"ID":"17883199-964b-4c64-a594-246fde087c55","Type":"ContainerStarted","Data":"9285ac63aba881761f25d9c80190b673afac68bc62b412d6f2ffc430b9e6e5df"} Nov 03 17:16:41.777280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.777256 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:41.823040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.822985 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" podStartSLOduration=1.822970399 podStartE2EDuration="1.822970399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:41.818856793 +0000 UTC m=+3697.152070362" watchObservedRunningTime="2026-11-03 17:16:41.822970399 +0000 UTC m=+3697.156184005" Nov 03 17:16:41.840851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.840809 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r"] Nov 03 17:16:41.846698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.846672 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.866644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.866601 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r"] Nov 03 17:16:41.868301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.868275 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:16:41.868472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.868364 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:16:41.868540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.868490 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:16:41.868598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.868584 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:16:41.872493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.872415 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" podStartSLOduration=1.8723982449999999 podStartE2EDuration="1.872398245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:41.86792801 +0000 UTC m=+3697.201141580" watchObservedRunningTime="2026-11-03 17:16:41.872398245 +0000 UTC m=+3697.205611820" Nov 03 17:16:41.882216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.882168 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.882533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.882506 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.882756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.882729 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4h45\" (UniqueName: \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.912224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.912184 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:16:41.917785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.917748 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.937327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.937292 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:16:41.968167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.968134 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9c4762-openshift-default-dockercfg-xfxf2\"" Nov 03 17:16:41.985522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985479 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.985719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985645 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.985719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985680 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.985857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985827 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.985996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985903 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.985996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985942 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.986074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.985999 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrkl\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.986074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986029 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.986142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986134 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4h45\" (UniqueName: \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.986177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986149 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:41.986218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986184 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.986277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986265 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.986434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.986412 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:41.988667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:41.988642 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:42.014359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.014322 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4h45\" (UniqueName: \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\") pod \"maas-api-revoke-sub-d0448f7a-z589r\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:42.148617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.143942 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.148617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.144096 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.148617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.144218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.148617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.144393 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.144431 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163174 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163188 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163241 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163278 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163321 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrkl\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.163830 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.164096 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.164173 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.164338 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.174474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.172241 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:42.199492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.199135 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrkl\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.215253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.213088 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.248313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.248274 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.259556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.248783 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\") pod \"e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.548584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.548539 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:42.737930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.737900 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r"] Nov 03 17:16:42.793005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.792981 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:42.793299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.793029 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" event={"ID":"579319fc-8242-4a15-98b0-bc0d3c23fb75","Type":"ContainerStarted","Data":"cb8650ee9a129851b023e05bb81b2b6ee7c54f704dd792c75e958ca8ceb2fafc"} Nov 03 17:16:42.866090 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:42.866057 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3529aefc_dbb5_415b_9192_b233f5cf4cba.slice/crio-71a01ddd0cec4ff30d25631feddbb1f8a823fcb5d89f51a5a39870704d1dadd0 WatchSource:0}: Error finding container 71a01ddd0cec4ff30d25631feddbb1f8a823fcb5d89f51a5a39870704d1dadd0: Status 404 returned error can't find the container with id 71a01ddd0cec4ff30d25631feddbb1f8a823fcb5d89f51a5a39870704d1dadd0 Nov 03 17:16:42.868019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.867994 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:16:42.869544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.869512 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:42.869629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.869614 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:42.869676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:42.869648 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:16:43.799735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:43.799694 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" event={"ID":"579319fc-8242-4a15-98b0-bc0d3c23fb75","Type":"ContainerStarted","Data":"fd7c5a7703eae9065527011fb1e33095b7df0bda440f86eb6c1a2adce782deac"} Nov 03 17:16:43.802038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:43.802004 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerStarted","Data":"9eaf88450cc28e923fc33028de10e815162fc3b49c8756cf93e8af1f223bf9fc"} Nov 03 17:16:43.802211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:43.802063 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerStarted","Data":"71a01ddd0cec4ff30d25631feddbb1f8a823fcb5d89f51a5a39870704d1dadd0"} Nov 03 17:16:43.849533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:43.849478 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" podStartSLOduration=2.849463048 podStartE2EDuration="2.849463048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:43.843238786 +0000 UTC m=+3699.176452355" watchObservedRunningTime="2026-11-03 17:16:43.849463048 +0000 UTC m=+3699.182676667" Nov 03 17:16:43.895429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:43.895359 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podStartSLOduration=2.8953399749999997 podStartE2EDuration="2.895339975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:43.890927095 +0000 UTC m=+3699.224140662" watchObservedRunningTime="2026-11-03 17:16:43.895339975 +0000 UTC m=+3699.228553547" Nov 03 17:16:44.549353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.549311 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:44.556596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.556564 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:44.813380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.813287 6001 generic.go:356] "Generic (PLEG): container finished" podID="579319fc-8242-4a15-98b0-bc0d3c23fb75" containerID="fd7c5a7703eae9065527011fb1e33095b7df0bda440f86eb6c1a2adce782deac" exitCode=0 Nov 03 17:16:44.813915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.813372 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" event={"ID":"579319fc-8242-4a15-98b0-bc0d3c23fb75","Type":"ContainerDied","Data":"fd7c5a7703eae9065527011fb1e33095b7df0bda440f86eb6c1a2adce782deac"} Nov 03 17:16:44.813915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.813758 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:44.814939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:44.814920 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:16:46.030981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.030950 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:46.071643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.071608 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\" (UniqueName: \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\") pod \"579319fc-8242-4a15-98b0-bc0d3c23fb75\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " Nov 03 17:16:46.071965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.071689 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\") pod \"579319fc-8242-4a15-98b0-bc0d3c23fb75\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " Nov 03 17:16:46.071965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.071715 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\") pod \"579319fc-8242-4a15-98b0-bc0d3c23fb75\" (UID: \"579319fc-8242-4a15-98b0-bc0d3c23fb75\") " Nov 03 17:16:46.072136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.072108 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca" pod "579319fc-8242-4a15-98b0-bc0d3c23fb75" (UID: "579319fc-8242-4a15-98b0-bc0d3c23fb75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:16:46.072587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.072536 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:46.074599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.074485 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45" pod "579319fc-8242-4a15-98b0-bc0d3c23fb75" (UID: "579319fc-8242-4a15-98b0-bc0d3c23fb75"). InnerVolumeSpecName "kube-api-access-z4h45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:46.075349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.075313 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca" pod "579319fc-8242-4a15-98b0-bc0d3c23fb75" (UID: "579319fc-8242-4a15-98b0-bc0d3c23fb75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:16:46.174381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.174345 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4h45\" (UniqueName: \"kubernetes.io/projected/579319fc-8242-4a15-98b0-bc0d3c23fb75-kube-api-access-z4h45\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:46.174381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.174379 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/579319fc-8242-4a15-98b0-bc0d3c23fb75-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:16:46.834152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.834123 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" Nov 03 17:16:46.834355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.834156 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r" event={"ID":"579319fc-8242-4a15-98b0-bc0d3c23fb75","Type":"ContainerDied","Data":"cb8650ee9a129851b023e05bb81b2b6ee7c54f704dd792c75e958ca8ceb2fafc"} Nov 03 17:16:46.834355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:46.834207 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb8650ee9a129851b023e05bb81b2b6ee7c54f704dd792c75e958ca8ceb2fafc" Nov 03 17:16:48.523470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.523040 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:16:48.543484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.524762 6001 cpu_manager.go:414] "containerMap: removing container" podUID="579319fc-8242-4a15-98b0-bc0d3c23fb75" containerName="revoke-keys" Nov 03 17:16:48.543484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.524786 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="579319fc-8242-4a15-98b0-bc0d3c23fb75" containerName="revoke-keys" Nov 03 17:16:48.543484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.525024 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="579319fc-8242-4a15-98b0-bc0d3c23fb75" containerName="revoke-keys" Nov 03 17:16:48.566438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.564587 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.594904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.582103 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c4762-dockercfg-5sp79\"" Nov 03 17:16:48.604482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.598240 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c4762\"" Nov 03 17:16:48.640526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.623100 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:16:48.673491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.673434 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhc6\" (UniqueName: \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.674201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.674173 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.776602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.776509 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhc6\" (UniqueName: \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.776770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.776706 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.777676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.777644 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.802356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.802187 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhc6\" (UniqueName: \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\") pod \"payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:48.925831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:48.925790 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:49.234843 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:49.234809 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a0f4c2_d6b3_41d2_bec3_29829c5bfe8f.slice/crio-f7d8f4bfbe8cc2918c3d33071a53042140d1f0f0ef07a0ceb343dedec6a3f81f WatchSource:0}: Error finding container f7d8f4bfbe8cc2918c3d33071a53042140d1f0f0ef07a0ceb343dedec6a3f81f: Status 404 returned error can't find the container with id f7d8f4bfbe8cc2918c3d33071a53042140d1f0f0ef07a0ceb343dedec6a3f81f Nov 03 17:16:49.237291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:49.237263 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:16:49.863198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:49.863157 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" event={"ID":"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f","Type":"ContainerStarted","Data":"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463"} Nov 03 17:16:49.863703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:49.863222 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" event={"ID":"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f","Type":"ContainerStarted","Data":"f7d8f4bfbe8cc2918c3d33071a53042140d1f0f0ef07a0ceb343dedec6a3f81f"} Nov 03 17:16:49.863703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:49.863253 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:16:49.914716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:49.914654 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" podStartSLOduration=1.9146284690000002 podStartE2EDuration="1.914628469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:49.913598229 +0000 UTC m=+3705.246811800" watchObservedRunningTime="2026-11-03 17:16:49.914628469 +0000 UTC m=+3705.247842041" Nov 03 17:16:52.255260 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:52.255217 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:16:53.809496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:53.808134 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:16:53.821472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:53.810112 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:16:55.652853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.652816 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:16:55.658940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.658916 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.679177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.679144 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:16:55.744052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.744013 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.744237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.744169 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.744371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.744355 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.744554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.744531 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkj9c\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.846023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.845988 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkj9c\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.846237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.846192 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.846360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.846336 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.846676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.846650 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.846835 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:55.846817 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-dea1e9ecdfa6: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:55.846947 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:55.846839 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:55.846947 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:55.846910 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials podName:fbdc070b-80b5-49b5-adff-be91808c1761 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:56.34689088 +0000 UTC m=+3711.680104443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761") : secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:55.847294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.847253 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.848404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.848380 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.849309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.849273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.871230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.871198 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkj9c\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:55.975591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:55.975564 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:56.284925 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:16:56.284892 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e12134_e6aa_4642_86e9_988de1449a76.slice/crio-ae3704ca9be6bd8a50056b9aaee44f7560efeb0c0a65bc3e6d01d184c4086346 WatchSource:0}: Error finding container ae3704ca9be6bd8a50056b9aaee44f7560efeb0c0a65bc3e6d01d184c4086346: Status 404 returned error can't find the container with id ae3704ca9be6bd8a50056b9aaee44f7560efeb0c0a65bc3e6d01d184c4086346 Nov 03 17:16:56.287296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:56.287264 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:16:56.365395 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:56.365353 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:56.365643 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:56.365627 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-dea1e9ecdfa6: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:56.365741 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:56.365730 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm: [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:56.365928 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:56.365913 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials podName:fbdc070b-80b5-49b5-adff-be91808c1761 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.365886992 +0000 UTC m=+3712.699100556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761") : [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:56.985847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:56.985769 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:16:57.024551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.024513 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" event={"ID":"50e12134-e6aa-4642-86e9-988de1449a76","Type":"ContainerStarted","Data":"1866a1b5554146ef037b93fc54aaf4ee83c333e912a8ec07fe03cec15b778a59"} Nov 03 17:16:57.024751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.024576 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" event={"ID":"50e12134-e6aa-4642-86e9-988de1449a76","Type":"ContainerStarted","Data":"ae3704ca9be6bd8a50056b9aaee44f7560efeb0c0a65bc3e6d01d184c4086346"} Nov 03 17:16:57.024751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.024671 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:16:57.089827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.089757 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" podStartSLOduration=2.089738475 podStartE2EDuration="2.089738475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:57.080255463 +0000 UTC m=+3712.413469034" watchObservedRunningTime="2026-11-03 17:16:57.089738475 +0000 UTC m=+3712.422952045" Nov 03 17:16:57.094769 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.094736 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:57.094769 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.094767 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69: secret "agc-cred-464294770e39" not found Nov 03 17:16:57.094998 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.094833 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.594814833 +0000 UTC m=+3712.928028394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : secret "agc-cred-464294770e39" not found Nov 03 17:16:57.410426 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.410329 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:57.410426 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.410369 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-dea1e9ecdfa6: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:57.410426 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.410381 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm: [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:57.410755 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.410458 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials podName:fbdc070b-80b5-49b5-adff-be91808c1761 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:59.410423272 +0000 UTC m=+3714.743636819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761") : [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:57.622332 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.622281 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:57.622615 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.622599 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69: secret "agc-cred-464294770e39" not found Nov 03 17:16:57.622812 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:57.622800 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:58.622777536 +0000 UTC m=+3713.955991097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : secret "agc-cred-464294770e39" not found Nov 03 17:16:57.974533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.974493 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:16:57.980283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:16:57.980248 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:16:58.133126 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.133090 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-plugins-e2e-extmod-a-da409b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:16:58.133544 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.133318 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:58.633289806 +0000 UTC m=+3713.966503362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:16:58.662014 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.661973 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:58.662014 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.662001 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69: secret "agc-cred-464294770e39" not found Nov 03 17:16:58.662247 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.662054 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:00.662039206 +0000 UTC m=+3715.995252752 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : secret "agc-cred-464294770e39" not found Nov 03 17:16:58.662247 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.662148 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-plugins-e2e-extmod-a-da409b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:16:58.662247 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:58.662224 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:16:59.662207782 +0000 UTC m=+3714.995421348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:16:59.496900 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.496871 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:16:59.496900 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.496897 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-dea1e9ecdfa6: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:16:59.496900 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.496907 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm: [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:59.497419 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.496960 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials podName:fbdc070b-80b5-49b5-adff-be91808c1761 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:03.49694655 +0000 UTC m=+3718.830160102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761") : [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:16:59.700716 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.700676 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-plugins-e2e-extmod-a-da409b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:16:59.700920 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:16:59.700767 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:01.70074223 +0000 UTC m=+3717.033955796 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:17:00.718752 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:00.718653 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:17:00.718752 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:00.718746 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69: secret "agc-cred-464294770e39" not found Nov 03 17:17:00.719273 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:00.718800 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:04.718785647 +0000 UTC m=+3720.051999195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : secret "agc-cred-464294770e39" not found Nov 03 17:17:00.899636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:00.899601 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:17:01.738879 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:01.738841 6001 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-plugins-e2e-extmod-a-da409b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:17:01.739301 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:01.738919 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume podName:50e12134-e6aa-4642-86e9-988de1449a76 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:05.738903465 +0000 UTC m=+3721.072117037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" (UID: "50e12134-e6aa-4642-86e9-988de1449a76") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-da409b" not found Nov 03 17:17:01.987154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:01.987103 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" podUID="532c40e8-cea0-4c1c-81e1-6fa228e01245" containerName="payload-processing" containerID="cri-o://041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd" gracePeriod=25 Nov 03 17:17:02.327875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.327845 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:17:02.453386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.453349 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\") pod \"532c40e8-cea0-4c1c-81e1-6fa228e01245\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " Nov 03 17:17:02.453632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.453610 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\") pod \"532c40e8-cea0-4c1c-81e1-6fa228e01245\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " Nov 03 17:17:02.453802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.453782 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\") pod \"532c40e8-cea0-4c1c-81e1-6fa228e01245\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " Nov 03 17:17:02.453870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.453845 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\") pod \"532c40e8-cea0-4c1c-81e1-6fa228e01245\" (UID: \"532c40e8-cea0-4c1c-81e1-6fa228e01245\") " Nov 03 17:17:02.454295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.454243 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume" pod "532c40e8-cea0-4c1c-81e1-6fa228e01245" (UID: "532c40e8-cea0-4c1c-81e1-6fa228e01245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:02.454295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.454263 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume" pod "532c40e8-cea0-4c1c-81e1-6fa228e01245" (UID: "532c40e8-cea0-4c1c-81e1-6fa228e01245"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:02.454700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.454680 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:02.454803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.454703 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/532c40e8-cea0-4c1c-81e1-6fa228e01245-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:02.455968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.455933 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls" pod "532c40e8-cea0-4c1c-81e1-6fa228e01245" (UID: "532c40e8-cea0-4c1c-81e1-6fa228e01245"). InnerVolumeSpecName "kube-api-access-jtsls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:02.456106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.456083 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials" pod "532c40e8-cea0-4c1c-81e1-6fa228e01245" (UID: "532c40e8-cea0-4c1c-81e1-6fa228e01245"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:02.575539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.572838 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:02.575539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.572879 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsls\" (UniqueName: \"kubernetes.io/projected/532c40e8-cea0-4c1c-81e1-6fa228e01245-kube-api-access-jtsls\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:02.974917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:02.974880 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" podUID="fbdc070b-80b5-49b5-adff-be91808c1761" containerName="payload-processing" containerID="cri-o://4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a" gracePeriod=25 Nov 03 17:17:03.238162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.236618 6001 generic.go:356] "Generic (PLEG): container finished" podID="532c40e8-cea0-4c1c-81e1-6fa228e01245" containerID="041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd" exitCode=0 Nov 03 17:17:03.238162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.236727 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" event={"ID":"532c40e8-cea0-4c1c-81e1-6fa228e01245","Type":"ContainerDied","Data":"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd"} Nov 03 17:17:03.238162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.236779 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" event={"ID":"532c40e8-cea0-4c1c-81e1-6fa228e01245","Type":"ContainerDied","Data":"728e44c791e4c54032ff45ec9c711c10bca32a625b27deb8e7505ba20a940213"} Nov 03 17:17:03.238162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.236812 6001 scope.go:122] "RemoveContainer" containerID="041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd" Nov 03 17:17:03.238162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.236997 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k" Nov 03 17:17:03.334120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.333081 6001 scope.go:122] "RemoveContainer" containerID="041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd" Nov 03 17:17:03.336773 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:03.335214 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd\": container with ID starting with 041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd not found: ID does not exist" containerID="041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd" Nov 03 17:17:03.336773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.335257 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd"} err="failed to get container status \"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd\": rpc error: code = NotFound desc = could not find container \"041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd\": container with ID starting with 041c334962b73279a7e5423b95cd90d0ec47093b82f166f6e431696ea56c10fd not found: ID does not exist" Nov 03 17:17:03.351055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.350974 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:17:03.393581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.393539 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-754ddcs2k"] Nov 03 17:17:03.519293 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:03.519187 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-464294770e39: secret "agc-cred-464294770e39" not found Nov 03 17:17:03.519293 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:03.519228 6001 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-da409b/agc-cred-dea1e9ecdfa6: secret "agc-cred-dea1e9ecdfa6" not found Nov 03 17:17:03.519293 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:03.519241 6001 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm: [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:17:03.519680 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:03.519310 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials podName:fbdc070b-80b5-49b5-adff-be91808c1761 nodeName:}" failed. No retries permitted until 2026-11-03 17:17:11.519288268 +0000 UTC m=+3726.852501817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761") : [secret "agc-cred-464294770e39" not found, secret "agc-cred-dea1e9ecdfa6" not found] Nov 03 17:17:03.581496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.579530 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:17:03.620290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.620249 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\") pod \"fbdc070b-80b5-49b5-adff-be91808c1761\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " Nov 03 17:17:03.620592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.620565 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\") pod \"fbdc070b-80b5-49b5-adff-be91808c1761\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " Nov 03 17:17:03.620793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.620753 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\") pod \"fbdc070b-80b5-49b5-adff-be91808c1761\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " Nov 03 17:17:03.620902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.620833 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\") pod \"fbdc070b-80b5-49b5-adff-be91808c1761\" (UID: \"fbdc070b-80b5-49b5-adff-be91808c1761\") " Nov 03 17:17:03.621644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.621289 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume" pod "fbdc070b-80b5-49b5-adff-be91808c1761" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:03.621644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.621495 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume" pod "fbdc070b-80b5-49b5-adff-be91808c1761" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:03.623935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.622133 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.623935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.622158 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbdc070b-80b5-49b5-adff-be91808c1761-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.624844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.624164 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials" pod "fbdc070b-80b5-49b5-adff-be91808c1761" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:03.635141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.635099 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2" pod "fbdc070b-80b5-49b5-adff-be91808c1761" (UID: "fbdc070b-80b5-49b5-adff-be91808c1761"). InnerVolumeSpecName "kube-api-access-dt9k2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:03.724360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.724319 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt9k2\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-kube-api-access-dt9k2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.724360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:03.724359 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbdc070b-80b5-49b5-adff-be91808c1761-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:04.023471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.022533 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:17:04.053475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.052519 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" podUID="50e12134-e6aa-4642-86e9-988de1449a76" containerName="payload-processing" containerID="cri-o://1866a1b5554146ef037b93fc54aaf4ee83c333e912a8ec07fe03cec15b778a59" gracePeriod=25 Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056265 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fbdc070b-80b5-49b5-adff-be91808c1761" containerName="payload-processing" Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056309 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdc070b-80b5-49b5-adff-be91808c1761" containerName="payload-processing" Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056495 6001 cpu_manager.go:414] "containerMap: removing container" podUID="532c40e8-cea0-4c1c-81e1-6fa228e01245" containerName="payload-processing" Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056507 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532c40e8-cea0-4c1c-81e1-6fa228e01245" containerName="payload-processing" Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056856 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="532c40e8-cea0-4c1c-81e1-6fa228e01245" containerName="payload-processing" Nov 03 17:17:04.064467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.056883 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdc070b-80b5-49b5-adff-be91808c1761" containerName="payload-processing" Nov 03 17:17:04.086782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.086747 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:17:04.087126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.087106 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.109087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.109054 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-547925\"" Nov 03 17:17:04.300488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.300059 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.300488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.300106 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-547925-dockercfg-hxlrd\"" Nov 03 17:17:04.300488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.300282 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8nsq\" (UniqueName: \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.315014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.314002 6001 generic.go:356] "Generic (PLEG): container finished" podID="fbdc070b-80b5-49b5-adff-be91808c1761" containerID="4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a" exitCode=0 Nov 03 17:17:04.315014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.314111 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" event={"ID":"fbdc070b-80b5-49b5-adff-be91808c1761","Type":"ContainerDied","Data":"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a"} Nov 03 17:17:04.315014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.314151 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" event={"ID":"fbdc070b-80b5-49b5-adff-be91808c1761","Type":"ContainerDied","Data":"b448d61f2a442234d059addad435bc6401da6d7ff25c8b1b70b266ce1ec91892"} Nov 03 17:17:04.315014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.314172 6001 scope.go:122] "RemoveContainer" containerID="4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a" Nov 03 17:17:04.315014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.314262 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm" Nov 03 17:17:04.322115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.322087 6001 generic.go:356] "Generic (PLEG): container finished" podID="50e12134-e6aa-4642-86e9-988de1449a76" containerID="1866a1b5554146ef037b93fc54aaf4ee83c333e912a8ec07fe03cec15b778a59" exitCode=0 Nov 03 17:17:04.322246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.322189 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" event={"ID":"50e12134-e6aa-4642-86e9-988de1449a76","Type":"ContainerDied","Data":"1866a1b5554146ef037b93fc54aaf4ee83c333e912a8ec07fe03cec15b778a59"} Nov 03 17:17:04.337890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.337823 6001 scope.go:122] "RemoveContainer" containerID="4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a" Nov 03 17:17:04.339410 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:04.339346 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a\": container with ID starting with 4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a not found: ID does not exist" containerID="4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a" Nov 03 17:17:04.339410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.339383 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a"} err="failed to get container status \"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a\": rpc error: code = NotFound desc = could not find container \"4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a\": container with ID starting with 4a8ccdd88453165c4260d041f5e1f27750cfeab3f540a851df12562f2194229a not found: ID does not exist" Nov 03 17:17:04.383508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.383465 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:17:04.391073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.391035 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-b76452lzm"] Nov 03 17:17:04.405390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.403376 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8nsq\" (UniqueName: \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.405390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.403953 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.405390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.405113 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.436523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.436487 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8nsq\" (UniqueName: \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\") pod \"payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.554721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.554631 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:17:04.622851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.622809 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:04.796526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.796489 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\") pod \"50e12134-e6aa-4642-86e9-988de1449a76\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " Nov 03 17:17:04.796757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.796586 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\") pod \"50e12134-e6aa-4642-86e9-988de1449a76\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " Nov 03 17:17:04.796842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.796824 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\") pod \"50e12134-e6aa-4642-86e9-988de1449a76\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " Nov 03 17:17:04.797018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.797003 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\") pod \"50e12134-e6aa-4642-86e9-988de1449a76\" (UID: \"50e12134-e6aa-4642-86e9-988de1449a76\") " Nov 03 17:17:04.800665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.800513 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume" pod "50e12134-e6aa-4642-86e9-988de1449a76" (UID: "50e12134-e6aa-4642-86e9-988de1449a76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:04.803156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.803111 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials" pod "50e12134-e6aa-4642-86e9-988de1449a76" (UID: "50e12134-e6aa-4642-86e9-988de1449a76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:04.803538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.803505 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume" pod "50e12134-e6aa-4642-86e9-988de1449a76" (UID: "50e12134-e6aa-4642-86e9-988de1449a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:04.826416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.825021 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c" pod "50e12134-e6aa-4642-86e9-988de1449a76" (UID: "50e12134-e6aa-4642-86e9-988de1449a76"). InnerVolumeSpecName "kube-api-access-kkj9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:04.844597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.844545 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532c40e8-cea0-4c1c-81e1-6fa228e01245" path="/var/lib/kubelet/pods/532c40e8-cea0-4c1c-81e1-6fa228e01245/volumes" Nov 03 17:17:04.845438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.845411 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdc070b-80b5-49b5-adff-be91808c1761" path="/var/lib/kubelet/pods/fbdc070b-80b5-49b5-adff-be91808c1761/volumes" Nov 03 17:17:04.940064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.939927 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkj9c\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-kube-api-access-kkj9c\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:04.940064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.939955 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/50e12134-e6aa-4642-86e9-988de1449a76-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:04.940064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.939972 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:04.940064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.939986 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50e12134-e6aa-4642-86e9-988de1449a76-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:17:04.943999 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:17:04.943959 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078a5cb0_78be_4650_9cdb_1dbaf17d2c67.slice/crio-e3a3c8ba3195770be4dbefc87d1f90fbe0ad347da417958a94639a9950d70368 WatchSource:0}: Error finding container e3a3c8ba3195770be4dbefc87d1f90fbe0ad347da417958a94639a9950d70368: Status 404 returned error can't find the container with id e3a3c8ba3195770be4dbefc87d1f90fbe0ad347da417958a94639a9950d70368 Nov 03 17:17:04.946734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:04.946705 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:17:05.388121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.388076 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" event={"ID":"078a5cb0-78be-4650-9cdb-1dbaf17d2c67","Type":"ContainerStarted","Data":"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf"} Nov 03 17:17:05.389002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.388972 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" event={"ID":"078a5cb0-78be-4650-9cdb-1dbaf17d2c67","Type":"ContainerStarted","Data":"e3a3c8ba3195770be4dbefc87d1f90fbe0ad347da417958a94639a9950d70368"} Nov 03 17:17:05.393465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.390319 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:05.447526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.443310 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" event={"ID":"50e12134-e6aa-4642-86e9-988de1449a76","Type":"ContainerDied","Data":"ae3704ca9be6bd8a50056b9aaee44f7560efeb0c0a65bc3e6d01d184c4086346"} Nov 03 17:17:05.447807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.447781 6001 scope.go:122] "RemoveContainer" containerID="1866a1b5554146ef037b93fc54aaf4ee83c333e912a8ec07fe03cec15b778a59" Nov 03 17:17:05.448095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.448078 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69" Nov 03 17:17:05.495785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.495694 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" podStartSLOduration=2.49567269 podStartE2EDuration="2.49567269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:05.439722452 +0000 UTC m=+3720.772936023" watchObservedRunningTime="2026-11-03 17:17:05.49567269 +0000 UTC m=+3720.828886260" Nov 03 17:17:05.723075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.722755 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:17:05.724785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.724757 6001 cpu_manager.go:414] "containerMap: removing container" podUID="50e12134-e6aa-4642-86e9-988de1449a76" containerName="payload-processing" Nov 03 17:17:05.724959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.724788 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50e12134-e6aa-4642-86e9-988de1449a76" containerName="payload-processing" Nov 03 17:17:05.726421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.726255 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50e12134-e6aa-4642-86e9-988de1449a76" containerName="payload-processing" Nov 03 17:17:05.739163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.739122 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:17:05.739347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.739247 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.748027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.747997 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:17:05.758299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.758271 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696bjqj69"] Nov 03 17:17:05.807133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807100 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807170 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stp6n\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807342 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807382 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807400 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807536 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807600 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807704 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.807768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.807744 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.810257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.810241 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-472bee-openshift-default-dockercfg-hqfr6\"" Nov 03 17:17:05.909313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909275 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909322 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909342 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909384 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909467 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909567 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909594 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909762 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909811 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.909867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909840 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.910028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909887 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.910028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.909897 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stp6n\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.910156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.910139 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.910297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.910274 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.912140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.912119 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.912379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.912358 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.934269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.934228 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:05.934432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:05.934245 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stp6n\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\") pod \"e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:06.066168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.062082 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:06.402141 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:17:06.402096 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37786abd_37ec_4fda_8e9b_a14e97a0711c.slice/crio-b455c7a7332a2586b9d7c7a72d1aebc7720892ed4b30b2caf75fba86b56f1dac WatchSource:0}: Error finding container b455c7a7332a2586b9d7c7a72d1aebc7720892ed4b30b2caf75fba86b56f1dac: Status 404 returned error can't find the container with id b455c7a7332a2586b9d7c7a72d1aebc7720892ed4b30b2caf75fba86b56f1dac Nov 03 17:17:06.404428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.404391 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:17:06.405749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.405714 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:17:06.405834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.405785 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:17:06.405834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.405818 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:17:06.455083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.455045 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerStarted","Data":"b455c7a7332a2586b9d7c7a72d1aebc7720892ed4b30b2caf75fba86b56f1dac"} Nov 03 17:17:06.793562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:06.793519 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50e12134-e6aa-4642-86e9-988de1449a76" path="/var/lib/kubelet/pods/50e12134-e6aa-4642-86e9-988de1449a76/volumes" Nov 03 17:17:07.467698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:07.467649 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerStarted","Data":"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591"} Nov 03 17:17:07.531537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:07.531425 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podStartSLOduration=2.531405092 podStartE2EDuration="2.531405092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:07.528718271 +0000 UTC m=+3722.861931859" watchObservedRunningTime="2026-11-03 17:17:07.531405092 +0000 UTC m=+3722.864618662" Nov 03 17:17:08.062650 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:08.062607 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:08.070225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:08.070197 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:08.477775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:08.477748 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:08.478871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:08.478851 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:17:11.037583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:11.037548 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl"] Nov 03 17:17:11.049948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:11.049917 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daad1b09-2sfdl"] Nov 03 17:17:12.253463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:12.216756 6001 scope.go:122] "RemoveContainer" containerID="84ca818aa2cae79b15564b246d38e8e61280e09d2d514e7e3291edd987947c04" Nov 03 17:17:12.423166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:12.423137 6001 scope.go:122] "RemoveContainer" containerID="2b62e0a7117071872d4e2c4fb3c6936365780cf8023e5e544542422be759b7c8" Nov 03 17:17:12.793863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:12.793828 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2427a2-7d72-4075-abe8-eff3d24d54fc" path="/var/lib/kubelet/pods/0d2427a2-7d72-4075-abe8-eff3d24d54fc/volumes" Nov 03 17:17:17.471349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:17:17.471319 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:17:22.253121 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:22.253080 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:17:52.527205 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:17:52.527165 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:18:06.706201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:06.706161 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:18:06.706915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:06.706226 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:18:09.231486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.231452 6001 generic.go:356] "Generic (PLEG): container finished" podID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerID="c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c" exitCode=0 Nov 03 17:18:09.231967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.231527 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerDied","Data":"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c"} Nov 03 17:18:09.232184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.232165 6001 scope.go:122] "RemoveContainer" containerID="c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c" Nov 03 17:18:09.234734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.234704 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:09.234827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.234779 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:09.234874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:09.234829 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:10.076332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.057788 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": read tcp 10.134.0.2:43034->10.134.1.94:15021: read: connection reset by peer" start-of-body= Nov 03 17:18:10.076332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.057853 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": read tcp 10.134.0.2:43034->10.134.1.94:15021: read: connection reset by peer" Nov 03 17:18:10.242606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.242573 6001 generic.go:356] "Generic (PLEG): container finished" podID="03a1743a-fe09-4413-87db-42f5e11da91c" containerID="e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697" exitCode=0 Nov 03 17:18:10.243050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.242654 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerDied","Data":"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697"} Nov 03 17:18:10.243212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.243196 6001 scope.go:122] "RemoveContainer" containerID="e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697" Nov 03 17:18:10.261005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.257837 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerStarted","Data":"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde"} Nov 03 17:18:10.265476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.265004 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:10.265476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.265117 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:10.265476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.265192 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:10.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.572016 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:18:10.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.572159 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:10.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:10.572207 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:11.299540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.299497 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerStarted","Data":"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90"} Nov 03 17:18:11.385470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.385032 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:18:11.385470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.385165 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:11.385470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.385214 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:11.573534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.570681 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:11.573534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:11.570750 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:12.386469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:12.383981 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:12.386469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:12.384067 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:12.498980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:12.498950 6001 scope.go:122] "RemoveContainer" containerID="480f25f2719cf0aff4afe77a5e683b84a31ade00a60a3eafe40b4f7cf3a7ab3d" Nov 03 17:18:12.574041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:12.574005 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:12.574335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:12.574306 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:13.387466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:13.387137 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:13.387466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:13.387238 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:13.578471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:13.574925 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:13.578471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:13.575012 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:14.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:14.383889 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:14.384470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:14.383953 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:14.571468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:14.570696 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:14.571468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:14.570769 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:15.042551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.042520 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt"] Nov 03 17:18:15.103840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.103800 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bbce2f2-4m6mt"] Nov 03 17:18:15.387037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.386931 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:15.387367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.387332 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:15.571133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.571091 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:18:15.571373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:15.571157 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:18:16.384496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.384369 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:18:16.384496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.384428 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:18:16.575771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.575734 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:18:16.576165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.576139 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:18:16.576883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.576864 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:18:16.809625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:16.809566 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe05aedd-b9d8-4def-8979-66031269a3bf" path="/var/lib/kubelet/pods/fe05aedd-b9d8-4def-8979-66031269a3bf/volumes" Nov 03 17:18:17.388790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:17.388759 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:18:17.389205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:17.389184 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:18:17.389853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:17.389833 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:18:22.252351 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:18:22.252316 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:18:25.033087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:25.033053 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz"] Nov 03 17:18:25.044854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:25.044826 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4434452-z2bbz"] Nov 03 17:18:26.792652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:26.792617 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e65447-f0ef-45f2-8b98-e0d2790e4fcc" path="/var/lib/kubelet/pods/87e65447-f0ef-45f2-8b98-e0d2790e4fcc/volumes" Nov 03 17:18:44.815205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:44.815171 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:44.815710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:44.815230 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:45.016492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.016429 6001 generic.go:356] "Generic (PLEG): container finished" podID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerID="9eaf88450cc28e923fc33028de10e815162fc3b49c8756cf93e8af1f223bf9fc" exitCode=0 Nov 03 17:18:45.016685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.016509 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerDied","Data":"9eaf88450cc28e923fc33028de10e815162fc3b49c8756cf93e8af1f223bf9fc"} Nov 03 17:18:45.016959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.016944 6001 scope.go:122] "RemoveContainer" containerID="9eaf88450cc28e923fc33028de10e815162fc3b49c8756cf93e8af1f223bf9fc" Nov 03 17:18:45.019069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.019037 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:45.019137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.019120 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:45.019175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:45.019162 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:18:46.027835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:46.027795 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerStarted","Data":"bb3f77f1684cb201ac8829820cf1aaac6fa2446a86e69fb3e20786377b8d6ab3"} Nov 03 17:18:46.551478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:46.550856 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:18:46.551478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:46.550985 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:46.551478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:46.551068 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:47.557850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:47.557813 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:47.558493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:47.558434 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:48.564465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:48.562753 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:48.564465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:48.562824 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:49.568472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:49.561659 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:49.568472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:49.561724 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:50.565955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:50.565912 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:50.566262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:50.566228 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:51.550735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:51.550688 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" start-of-body= Nov 03 17:18:51.551412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:51.551352 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.102:15021/healthz/ready\": dial tcp 10.134.1.102:15021: connect: connection refused" Nov 03 17:18:52.253361 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:18:52.253321 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:18:52.555176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:52.555086 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:18:52.555654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:52.555631 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:18:52.556504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:52.556481 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:18:56.057291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:56.057250 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s"] Nov 03 17:18:56.081942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:56.081904 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f210497d-xg64s"] Nov 03 17:18:56.834682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:18:56.834644 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70b7386-efc5-4549-bd6a-461ec3e24c21" path="/var/lib/kubelet/pods/b70b7386-efc5-4549-bd6a-461ec3e24c21/volumes" Nov 03 17:19:06.032192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:06.032150 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn"] Nov 03 17:19:06.044667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:06.044628 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaff300f-72qvn"] Nov 03 17:19:06.841510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:06.840724 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73810920-f5ae-4a5b-bebc-77df1898b031" path="/var/lib/kubelet/pods/73810920-f5ae-4a5b-bebc-77df1898b031/volumes" Nov 03 17:19:08.664566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.664521 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:19:08.691900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.691867 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.703954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.703913 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:19:08.710021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.709993 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89df9a-openshift-default-dockercfg-hp8vk\"" Nov 03 17:19:08.766922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.748174 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.766922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.766766 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.766922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.766871 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.766940 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.767071 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctg6b\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.767144 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.767275 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.767318 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.767566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.767527 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925248 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925383 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctg6b\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925458 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925557 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925594 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925740 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.925839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925800 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.926057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925885 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.926057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.925961 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.978476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.926574 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.998497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.992976 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.998497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.993814 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:08.998497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.997302 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.004305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:08.999120 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.037386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.037311 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.058141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.058070 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.151607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.138383 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.162147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.162091 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctg6b\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.197526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.193154 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7"] Nov 03 17:19:09.212707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.202082 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e95d963b-c7nq7"] Nov 03 17:19:09.369301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.368828 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:09.701644 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:19:09.701609 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70deee7_57c3_46ff_bebc_0ba1bec16c11.slice/crio-cb09c518e39319d4eb48c6f27c97e94e4a4e78ab49e51f042d3d5cb64172d094 WatchSource:0}: Error finding container cb09c518e39319d4eb48c6f27c97e94e4a4e78ab49e51f042d3d5cb64172d094: Status 404 returned error can't find the container with id cb09c518e39319d4eb48c6f27c97e94e4a4e78ab49e51f042d3d5cb64172d094 Nov 03 17:19:09.704177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.704149 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:19:09.705894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.705850 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:09.705980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.705944 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:09.706026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:09.705983 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:10.422219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:10.422175 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerStarted","Data":"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e"} Nov 03 17:19:10.422417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:10.422265 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerStarted","Data":"cb09c518e39319d4eb48c6f27c97e94e4a4e78ab49e51f042d3d5cb64172d094"} Nov 03 17:19:10.484478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:10.484383 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podStartSLOduration=2.484366905 podStartE2EDuration="2.484366905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:10.482702755 +0000 UTC m=+3845.815916331" watchObservedRunningTime="2026-11-03 17:19:10.484366905 +0000 UTC m=+3845.817580473" Nov 03 17:19:10.793687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:10.793652 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe4815e-29e6-48c8-a035-ef1c00bc571c" path="/var/lib/kubelet/pods/4fe4815e-29e6-48c8-a035-ef1c00bc571c/volumes" Nov 03 17:19:11.370040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:11.370001 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:11.376978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:11.376936 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:11.432582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:11.432554 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:11.433726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:11.433703 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:19:12.533480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:12.533410 6001 scope.go:122] "RemoveContainer" containerID="8b2fbc8a5416955669bc14b55ed56770a151d0ca60c9d36b153d391a85a2ea46" Nov 03 17:19:15.535202 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.535168 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:19:15.786782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.786702 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:19:15.835893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.798624 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:15.852737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.852708 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:19:15.884995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.884685 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2h8\" (UniqueName: \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:15.885857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.885315 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:15.977937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.977888 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-89df9a\"" Nov 03 17:19:15.978188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.978055 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-89df9a-dockercfg-lsjpk\"" Nov 03 17:19:15.994468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.993177 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2h8\" (UniqueName: \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:15.994468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:15.993599 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:16.024403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.022864 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:16.030183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.030135 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:19:16.030378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.030274 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.044967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.044892 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2h8\" (UniqueName: \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\") pod \"payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:16.052482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.052407 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk"] Nov 03 17:19:16.058703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.058672 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e82e77e-5mjlk"] Nov 03 17:19:16.095681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.095642 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmvx\" (UniqueName: \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.095992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.095971 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.112097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.112062 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:16.197791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.197755 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmvx\" (UniqueName: \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.198697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.198570 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.199534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.199505 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.222847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.222814 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmvx\" (UniqueName: \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\") pod \"payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.480050 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:19:16.480019 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7990af_35df_4fee_b294_a20f641d6742.slice/crio-effea5aae53180d21039f7b2a9eb4623c7d9604145faa6a3172958e938d45613 WatchSource:0}: Error finding container effea5aae53180d21039f7b2a9eb4623c7d9604145faa6a3172958e938d45613: Status 404 returned error can't find the container with id effea5aae53180d21039f7b2a9eb4623c7d9604145faa6a3172958e938d45613 Nov 03 17:19:16.481586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.481557 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:19:16.491435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.491412 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:16.563910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.563860 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" event={"ID":"1d7990af-35df-4fee-b294-a20f641d6742","Type":"ContainerStarted","Data":"effea5aae53180d21039f7b2a9eb4623c7d9604145faa6a3172958e938d45613"} Nov 03 17:19:16.801355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.801258 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a161ffb3-a2bf-44c5-9727-f15d9109b8a4" path="/var/lib/kubelet/pods/a161ffb3-a2bf-44c5-9727-f15d9109b8a4/volumes" Nov 03 17:19:16.950261 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:19:16.950232 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954947d0_810d_49f4_b404_f6d36f01ecb9.slice/crio-99c44d2988b496df6c49707ef2480f01f8209639abdf7e83a6863b93c8f08699 WatchSource:0}: Error finding container 99c44d2988b496df6c49707ef2480f01f8209639abdf7e83a6863b93c8f08699: Status 404 returned error can't find the container with id 99c44d2988b496df6c49707ef2480f01f8209639abdf7e83a6863b93c8f08699 Nov 03 17:19:16.955286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:16.955256 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:19:17.578565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.578515 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" event={"ID":"1d7990af-35df-4fee-b294-a20f641d6742","Type":"ContainerStarted","Data":"399f977876b3440394bd2057319c7ecb5a5ef565036b89d35e09446b9e9e66fa"} Nov 03 17:19:17.579163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.578633 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:17.580845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.580818 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" event={"ID":"954947d0-810d-49f4-b404-f6d36f01ecb9","Type":"ContainerStarted","Data":"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f"} Nov 03 17:19:17.580988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.580863 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" event={"ID":"954947d0-810d-49f4-b404-f6d36f01ecb9","Type":"ContainerStarted","Data":"99c44d2988b496df6c49707ef2480f01f8209639abdf7e83a6863b93c8f08699"} Nov 03 17:19:17.580988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.580894 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:17.670940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.670876 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" podStartSLOduration=2.67085392 podStartE2EDuration="2.67085392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:17.66865038 +0000 UTC m=+3853.001863951" watchObservedRunningTime="2026-11-03 17:19:17.67085392 +0000 UTC m=+3853.004067490" Nov 03 17:19:17.674455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:17.674378 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" podStartSLOduration=2.674359874 podStartE2EDuration="2.674359874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:17.625585656 +0000 UTC m=+3852.958799225" watchObservedRunningTime="2026-11-03 17:19:17.674359874 +0000 UTC m=+3853.007573446" Nov 03 17:19:19.606575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.606539 6001 generic.go:356] "Generic (PLEG): container finished" podID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerID="1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591" exitCode=0 Nov 03 17:19:19.607047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.606610 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerDied","Data":"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591"} Nov 03 17:19:19.607183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.607167 6001 scope.go:122] "RemoveContainer" containerID="1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591" Nov 03 17:19:19.609629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.609593 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:19.609738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.609673 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:19.609738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:19.609714 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:20.619816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:20.619780 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerStarted","Data":"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e"} Nov 03 17:19:21.067467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:21.066559 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:19:21.072611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:21.072571 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:21.072812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:21.072644 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:22.078498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:22.069274 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:22.078498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:22.069350 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:22.277324 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:19:22.277272 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:19:23.063488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:23.062896 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:23.063488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:23.062988 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:23.485664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:23.485611 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:19:24.063624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:24.063588 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:24.063941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:24.063911 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:25.069470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:25.068721 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:25.069470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:25.068799 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:26.063469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:26.062877 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" start-of-body= Nov 03 17:19:26.063469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:26.062956 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.106:15021/healthz/ready\": dial tcp 10.134.1.106:15021: connect: connection refused" Nov 03 17:19:27.165281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:27.165239 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:19:27.167252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:27.167214 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:19:28.640938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:28.640903 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:19:28.642697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:28.642655 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:19:34.799222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.799188 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:19:34.805709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.805655 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.829185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.829153 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:19:34.833316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.833285 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-24cb6d-openshift-default-dockercfg-dz77t\"" Nov 03 17:19:34.978317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978276 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978474 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbzk6\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978533 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978758 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978844 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978875 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.978897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.978893 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.979049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.979030 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:34.979191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:34.979176 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.080950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.080864 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.080950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.080915 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.080950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.080949 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.080999 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081088 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081129 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081176 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbzk6\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081242 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081297 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081504 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081560 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081578 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.081791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.081770 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.082219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.082185 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.084021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.083999 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.084195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.084173 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.105676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.105644 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.105995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.105967 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbzk6\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.131082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.131042 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:35.514886 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:19:35.514854 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a00a1fe_17ef_4200_9d20_c999c2f92516.slice/crio-1b091ccf068ee5371ca793c05e2bd4fc683cb8783761d0d5b4eb6dfcbf3c990d WatchSource:0}: Error finding container 1b091ccf068ee5371ca793c05e2bd4fc683cb8783761d0d5b4eb6dfcbf3c990d: Status 404 returned error can't find the container with id 1b091ccf068ee5371ca793c05e2bd4fc683cb8783761d0d5b4eb6dfcbf3c990d Nov 03 17:19:35.515616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.515588 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:19:35.518414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.518381 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:35.518515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.518467 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:35.518515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.518499 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:35.992764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.992725 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" event={"ID":"6a00a1fe-17ef-4200-9d20-c999c2f92516","Type":"ContainerStarted","Data":"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05"} Nov 03 17:19:35.993295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:35.992793 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" event={"ID":"6a00a1fe-17ef-4200-9d20-c999c2f92516","Type":"ContainerStarted","Data":"1b091ccf068ee5371ca793c05e2bd4fc683cb8783761d0d5b4eb6dfcbf3c990d"} Nov 03 17:19:36.037976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.037945 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7"] Nov 03 17:19:36.046659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.046624 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daee7470-m6hg7"] Nov 03 17:19:36.058897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.058845 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podStartSLOduration=2.058830806 podStartE2EDuration="2.058830806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:36.054992729 +0000 UTC m=+3871.388206303" watchObservedRunningTime="2026-11-03 17:19:36.058830806 +0000 UTC m=+3871.392044375" Nov 03 17:19:36.132239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.132197 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:36.133897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.133870 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:36.133997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.133926 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:36.795622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:36.795571 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717b27ad-70d0-4b5d-a806-164ef4d28c1a" path="/var/lib/kubelet/pods/717b27ad-70d0-4b5d-a806-164ef4d28c1a/volumes" Nov 03 17:19:37.131997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:37.131896 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:37.131997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:37.131957 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:38.132263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:38.132221 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:38.132733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:38.132282 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:39.131775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:39.131732 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:39.131985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:39.131804 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:40.033496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:40.033454 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758"] Nov 03 17:19:40.066468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:40.066202 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eeb0c48-zz758"] Nov 03 17:19:40.131755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:40.131711 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:40.131962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:40.131780 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:40.793290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:40.793250 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134fcd64-e2c1-4306-8a25-287f9e1572c1" path="/var/lib/kubelet/pods/134fcd64-e2c1-4306-8a25-287f9e1572c1/volumes" Nov 03 17:19:41.131575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:41.131429 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:41.131575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:41.131550 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:42.132232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:42.132197 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" start-of-body= Nov 03 17:19:42.132684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:42.132254 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.110:15021/healthz/ready\": dial tcp 10.134.1.110:15021: connect: connection refused" Nov 03 17:19:43.137489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:43.137426 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:43.137975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:43.137959 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:43.138703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:43.138677 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:19:47.034316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:47.034226 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48"] Nov 03 17:19:47.046864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:47.046832 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f558d86-prd48"] Nov 03 17:19:48.796239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:48.796201 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70a7aee-b98e-412b-a659-d8d1f4a3fc1b" path="/var/lib/kubelet/pods/b70a7aee-b98e-412b-a659-d8d1f4a3fc1b/volumes" Nov 03 17:19:51.108235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.108191 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:19:51.114294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.114268 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.169983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.169951 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fc40cc-openshift-default-dockercfg-hbqxj\"" Nov 03 17:19:51.176985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.176947 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:19:51.210683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.210640 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.210905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.210766 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.210905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.210800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.210910 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.210966 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.211111 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lp2j\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.211222 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.211436 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.211712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.211680 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313368 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313602 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313657 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313760 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313823 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313848 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.313965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313942 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.314364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.313987 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.314364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.314036 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lp2j\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.314364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.314246 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.314745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.314707 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.315393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.314716 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.315393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.314935 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.315606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.315320 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.317385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.317360 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.317703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.317683 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.338339 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.338298 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.338540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.338373 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lp2j\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.450170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.450072 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:51.879133 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:19:51.879107 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0818e36f_3d2a_4ab2_b48e_311247ba15fc.slice/crio-b5aa43a5934e8f17ba16d5ece285d281ac85523a3ded8b5b3827cd81cb967313 WatchSource:0}: Error finding container b5aa43a5934e8f17ba16d5ece285d281ac85523a3ded8b5b3827cd81cb967313: Status 404 returned error can't find the container with id b5aa43a5934e8f17ba16d5ece285d281ac85523a3ded8b5b3827cd81cb967313 Nov 03 17:19:51.879999 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.879968 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:19:51.882670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.882631 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:51.882791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.882718 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:51.882791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:51.882766 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:19:52.252680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.251696 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" event={"ID":"0818e36f-3d2a-4ab2-b48e-311247ba15fc","Type":"ContainerStarted","Data":"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a"} Nov 03 17:19:52.252680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.251764 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" event={"ID":"0818e36f-3d2a-4ab2-b48e-311247ba15fc","Type":"ContainerStarted","Data":"b5aa43a5934e8f17ba16d5ece285d281ac85523a3ded8b5b3827cd81cb967313"} Nov 03 17:19:52.256605 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:19:52.256564 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:19:52.313070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.313002 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podStartSLOduration=1.3129802289999999 podStartE2EDuration="1.312980229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:52.312163697 +0000 UTC m=+3887.645377267" watchObservedRunningTime="2026-11-03 17:19:52.312980229 +0000 UTC m=+3887.646193800" Nov 03 17:19:52.450875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.450838 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:52.452873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.452836 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:52.453041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:52.452899 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:53.473015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:53.472971 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:53.473816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:53.473776 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:54.465001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:54.464959 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:54.465326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:54.465292 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:54.587609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:54.587566 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 17:19:54.588167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:54.587794 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" containerID="cri-o://16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458" gracePeriod=30 Nov 03 17:19:55.457471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:55.455331 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:55.457471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:55.455395 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:56.454671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:56.454616 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:56.455374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:56.455335 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:57.461996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:57.461952 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:57.462763 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:57.462723 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:58.465469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:58.464974 6001 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" start-of-body= Nov 03 17:19:58.465469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:58.465066 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.111:15021/healthz/ready\": dial tcp 10.134.1.111:15021: connect: connection refused" Nov 03 17:19:59.143402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.143364 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:19:59.143672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.143641 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" containerID="cri-o://a51c1b2c6356c21ca77a573fab6f3abe7a19a5c78a217cd95f71a23c315e5356" gracePeriod=30 Nov 03 17:19:59.456329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.456242 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:59.456719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.456687 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:59.457309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.457288 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:19:59.636055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.636016 6001 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 17:19:59.636634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.636103 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 17:19:59.912961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:19:59.912930 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 17:20:00.010151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010118 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010164 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010295 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010324 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010460 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010543 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010606 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010786 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:00.010986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010856 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.010986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010942 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:00.011263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.010984 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:00.011263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.011066 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:00.011263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.011082 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:00.011263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.011160 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\") pod \"c1336b33-4119-41cf-b0bd-6458d1ead764\" (UID: \"c1336b33-4119-41cf-b0bd-6458d1ead764\") " Nov 03 17:20:00.012385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.012357 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1336b33-4119-41cf-b0bd-6458d1ead764-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.012547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.012389 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.012547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.012404 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.012547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.012419 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.012547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.012435 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.013261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.013237 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:00.013359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.013281 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:00.013421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.013401 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:00.013648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.013625 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq" pod "c1336b33-4119-41cf-b0bd-6458d1ead764" (UID: "c1336b33-4119-41cf-b0bd-6458d1ead764"). InnerVolumeSpecName "kube-api-access-jq7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:00.114591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.114560 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.114591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.114590 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.114811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.114602 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1336b33-4119-41cf-b0bd-6458d1ead764-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.114811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.114614 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7pq\" (UniqueName: \"kubernetes.io/projected/c1336b33-4119-41cf-b0bd-6458d1ead764-kube-api-access-jq7pq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:00.378942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.378855 6001 generic.go:356] "Generic (PLEG): container finished" podID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerID="16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458" exitCode=0 Nov 03 17:20:00.378942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.378904 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" Nov 03 17:20:00.379157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.378935 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" event={"ID":"c1336b33-4119-41cf-b0bd-6458d1ead764","Type":"ContainerDied","Data":"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458"} Nov 03 17:20:00.379157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.378991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn" event={"ID":"c1336b33-4119-41cf-b0bd-6458d1ead764","Type":"ContainerDied","Data":"f74c77964380b58c6775a8ceff2d6e54413edc554c5cd569b937f73d5960d469"} Nov 03 17:20:00.379157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.379014 6001 scope.go:122] "RemoveContainer" containerID="16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458" Nov 03 17:20:00.398167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.398143 6001 scope.go:122] "RemoveContainer" containerID="16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458" Nov 03 17:20:00.398807 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:00.398777 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458\": container with ID starting with 16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458 not found: ID does not exist" containerID="16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458" Nov 03 17:20:00.398992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.398964 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458"} err="failed to get container status \"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458\": rpc error: code = NotFound desc = could not find container \"16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458\": container with ID starting with 16d0d4b3370490334a84f3db20a310696c85fc99e3c606e2a4ffed3f758fe458 not found: ID does not exist" Nov 03 17:20:00.438329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.438289 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 17:20:00.449815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.449787 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-7bqwn"] Nov 03 17:20:00.795903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:00.795862 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" path="/var/lib/kubelet/pods/c1336b33-4119-41cf-b0bd-6458d1ead764/volumes" Nov 03 17:20:04.445796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.445693 6001 generic.go:356] "Generic (PLEG): container finished" podID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerID="a51c1b2c6356c21ca77a573fab6f3abe7a19a5c78a217cd95f71a23c315e5356" exitCode=0 Nov 03 17:20:04.445796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.445786 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" event={"ID":"8678db30-57d9-402a-9b9e-f05362e0ef2a","Type":"ContainerDied","Data":"a51c1b2c6356c21ca77a573fab6f3abe7a19a5c78a217cd95f71a23c315e5356"} Nov 03 17:20:04.508254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.508216 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:20:04.547210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547173 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547268 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547301 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547334 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547372 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547412 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547552 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547574 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547682 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547723 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547768 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:04.547779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547777 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\") pod \"8678db30-57d9-402a-9b9e-f05362e0ef2a\" (UID: \"8678db30-57d9-402a-9b9e-f05362e0ef2a\") " Nov 03 17:20:04.548118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.547939 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.548186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548133 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.548795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548764 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.548795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548796 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8678db30-57d9-402a-9b9e-f05362e0ef2a-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.548988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548809 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.548988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548824 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.548988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.548837 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.550153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.550129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "kube-api-access-4jjbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:04.550507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.550481 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:04.550766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.550740 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:04.550873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.550848 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy" pod "8678db30-57d9-402a-9b9e-f05362e0ef2a" (UID: "8678db30-57d9-402a-9b9e-f05362e0ef2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.651392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.651343 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.651392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.651383 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.651392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.651397 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8678db30-57d9-402a-9b9e-f05362e0ef2a-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.651754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:04.651410 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjbx\" (UniqueName: \"kubernetes.io/projected/8678db30-57d9-402a-9b9e-f05362e0ef2a-kube-api-access-4jjbx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:20:05.458806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.458761 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" event={"ID":"8678db30-57d9-402a-9b9e-f05362e0ef2a","Type":"ContainerDied","Data":"bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64"} Nov 03 17:20:05.459265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.458811 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" Nov 03 17:20:05.459265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.458849 6001 scope.go:122] "RemoveContainer" containerID="a51c1b2c6356c21ca77a573fab6f3abe7a19a5c78a217cd95f71a23c315e5356" Nov 03 17:20:05.491224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.491189 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:20:05.491408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.491256 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:20:05.523040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.523006 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:20:05.538009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.537977 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-v8qwn"] Nov 03 17:20:05.623846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.623816 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:20:05.740333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:05.740301 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:20:06.748036 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:06.747994 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:06.793226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:06.793191 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" path="/var/lib/kubelet/pods/8678db30-57d9-402a-9b9e-f05362e0ef2a/volumes" Nov 03 17:20:10.964364 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:10.964322 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:10.986916 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:10.986870 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:10.987627 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:10.987595 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:12.573489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:12.573437 6001 scope.go:122] "RemoveContainer" containerID="ed5e29a9ab1d0fa4c0d42648fb408bb3347eb0fb86e007b54e843df3a36e1301" Nov 03 17:20:12.587421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:12.587398 6001 scope.go:122] "RemoveContainer" containerID="7b99b304f650bdfe52a5e6de6615b8251a30acbdc80af39c149199b432d44d47" Nov 03 17:20:12.601800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:12.601774 6001 scope.go:122] "RemoveContainer" containerID="d595202e369f819e5b35b3c94220895c7b15af675f4de390309ff347c5dd5b1b" Nov 03 17:20:12.616297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:20:12.616222 6001 scope.go:122] "RemoveContainer" containerID="d6a9747fc0762aa5dfde0176d47615a971120a964a941703c57b7d8c0bb217be" Nov 03 17:20:21.086350 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:21.086312 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:21.644892 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:21.644845 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:22.254838 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:22.254802 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:20:31.205734 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:31.205691 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:36.655452 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:36.655393 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:41.229239 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:41.229192 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:51.353914 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:51.353875 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:51.638394 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:51.638288 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache]" Nov 03 17:20:52.252742 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:20:52.252702 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:21:01.512955 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:01.512912 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice/crio-bbcd68a4e7b7b065c1d3516634946e16bd868fa7c69ce94d1c3ae7c6c54a0b64\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8678db30_57d9_402a_9b9e_f05362e0ef2a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:21:02.737669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.737631 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:21:02.738104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.737853 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" containerID="cri-o://3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee" gracePeriod=30 Nov 03 17:21:02.781903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.781864 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:21:02.782155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.782125 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" containerID="cri-o://d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b" gracePeriod=30 Nov 03 17:21:02.805262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.805229 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:21:02.805509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:02.805474 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" containerID="cri-o://9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b" gracePeriod=30 Nov 03 17:21:08.183515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.180151 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:21:08.183515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.182659 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:21:08.329322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.329292 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:21:08.331856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.331821 6001 generic.go:356] "Generic (PLEG): container finished" podID="6d9b74e6-ff47-457e-8806-781259a6d687" containerID="9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b" exitCode=0 Nov 03 17:21:08.332056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.332035 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" event={"ID":"6d9b74e6-ff47-457e-8806-781259a6d687","Type":"ContainerDied","Data":"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b"} Nov 03 17:21:08.332104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.332095 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" event={"ID":"6d9b74e6-ff47-457e-8806-781259a6d687","Type":"ContainerDied","Data":"92c2cb66005f7ce3097e7f9025e03c9083846647ca185f7d7bfd3f5edcbeeda2"} Nov 03 17:21:08.332142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.332126 6001 scope.go:122] "RemoveContainer" containerID="9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b" Nov 03 17:21:08.332317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.332302 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln" Nov 03 17:21:08.337540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.335724 6001 generic.go:356] "Generic (PLEG): container finished" podID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerID="d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b" exitCode=0 Nov 03 17:21:08.337540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.335819 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" Nov 03 17:21:08.337540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.336019 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" event={"ID":"d5ab24eb-f971-4ff2-93c9-8a6539ed40db","Type":"ContainerDied","Data":"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b"} Nov 03 17:21:08.337540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.336069 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c" event={"ID":"d5ab24eb-f971-4ff2-93c9-8a6539ed40db","Type":"ContainerDied","Data":"0deb7e8585eb1e187499b12cca964b0568796389fb36423c66e68ae58e1f1d32"} Nov 03 17:21:08.345831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.345703 6001 generic.go:356] "Generic (PLEG): container finished" podID="a42248da-ec4d-4810-b331-250069ca99c8" containerID="3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee" exitCode=0 Nov 03 17:21:08.345958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.345836 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" event={"ID":"a42248da-ec4d-4810-b331-250069ca99c8","Type":"ContainerDied","Data":"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee"} Nov 03 17:21:08.345958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.345911 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" event={"ID":"a42248da-ec4d-4810-b331-250069ca99c8","Type":"ContainerDied","Data":"f55c72b2eeb2bf04d521f66a625ad9840cef60e22fa96d862a1b393585d26297"} Nov 03 17:21:08.346072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.346021 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql" Nov 03 17:21:08.351314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351285 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.351494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.351494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351361 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.351494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351391 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.351494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351422 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.351824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351803 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.351966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351948 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.352063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.351776 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:21:08.352130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352074 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.352238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352221 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.352424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352405 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.352645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352626 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.352755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352439 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.352838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352810 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.353009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.352991 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.353196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353072 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.353308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353286 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.353381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353352 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.353508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353491 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.353742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353685 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.353964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.353942 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.354123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354102 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.354265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354242 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.354373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354356 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.354524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354506 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.354738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354718 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.354875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354859 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.354997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.354982 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.355144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355127 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.355293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355278 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.355508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355489 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.355658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355641 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\") pod \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\" (UID: \"d5ab24eb-f971-4ff2-93c9-8a6539ed40db\") " Nov 03 17:21:08.355788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355772 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.355943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.355925 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\") pod \"a42248da-ec4d-4810-b331-250069ca99c8\" (UID: \"a42248da-ec4d-4810-b331-250069ca99c8\") " Nov 03 17:21:08.356073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.356054 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\") pod \"6d9b74e6-ff47-457e-8806-781259a6d687\" (UID: \"6d9b74e6-ff47-457e-8806-781259a6d687\") " Nov 03 17:21:08.357348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357318 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.357544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357529 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.357652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357641 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.357744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357731 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.357836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357824 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.357918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357907 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.358001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.357991 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.358097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358086 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.358190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358108 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.358349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358323 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.358526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358423 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.359647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358634 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.359647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.358838 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:21:08.359647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.359032 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:21:08.360304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.360276 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.360549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.360523 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.361927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.361898 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.363551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.363515 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:21:08.363657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.363598 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.365644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.365554 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.365644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.365603 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.365824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.365436 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:08.365992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.365966 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p" pod "d5ab24eb-f971-4ff2-93c9-8a6539ed40db" (UID: "d5ab24eb-f971-4ff2-93c9-8a6539ed40db"). InnerVolumeSpecName "kube-api-access-9ww5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.366145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.366126 6001 scope.go:122] "RemoveContainer" containerID="9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b" Nov 03 17:21:08.367046 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:08.366636 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b\": container with ID starting with 9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b not found: ID does not exist" containerID="9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b" Nov 03 17:21:08.367046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.366684 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b"} err="failed to get container status \"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b\": rpc error: code = NotFound desc = could not find container \"9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b\": container with ID starting with 9d7f4a382d487c2a6e6a29028e252c81836e1d6a597ed5da05fcd7225805297b not found: ID does not exist" Nov 03 17:21:08.367046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.366711 6001 scope.go:122] "RemoveContainer" containerID="d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b" Nov 03 17:21:08.367951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.367419 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:21:08.367951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.367462 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "kube-api-access-pk2xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.367951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.367740 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc" pod "6d9b74e6-ff47-457e-8806-781259a6d687" (UID: "6d9b74e6-ff47-457e-8806-781259a6d687"). InnerVolumeSpecName "kube-api-access-m95pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:08.368177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.368126 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo" pod "a42248da-ec4d-4810-b331-250069ca99c8" (UID: "a42248da-ec4d-4810-b331-250069ca99c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:21:08.407896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.407868 6001 scope.go:122] "RemoveContainer" containerID="d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b" Nov 03 17:21:08.408497 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:08.408317 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b\": container with ID starting with d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b not found: ID does not exist" containerID="d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b" Nov 03 17:21:08.408497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.408362 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b"} err="failed to get container status \"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b\": rpc error: code = NotFound desc = could not find container \"d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b\": container with ID starting with d7790fb1d17c4c86362b4d8d6138943b0d78a8943a701a63e1ca761b023a1e7b not found: ID does not exist" Nov 03 17:21:08.408497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.408384 6001 scope.go:122] "RemoveContainer" containerID="3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee" Nov 03 17:21:08.428567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.428542 6001 scope.go:122] "RemoveContainer" containerID="3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee" Nov 03 17:21:08.428975 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:08.428953 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee\": container with ID starting with 3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee not found: ID does not exist" containerID="3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee" Nov 03 17:21:08.429091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.428986 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee"} err="failed to get container status \"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee\": rpc error: code = NotFound desc = could not find container \"3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee\": container with ID starting with 3a61ded370052178833086bb6742b1f8be61ea02a11c223f6cfe4838f7ba24ee not found: ID does not exist" Nov 03 17:21:08.460978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.460924 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.460978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.460971 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.460989 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461004 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461018 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461030 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461042 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a42248da-ec4d-4810-b331-250069ca99c8-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461055 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461066 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9b74e6-ff47-457e-8806-781259a6d687-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461079 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461094 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9b74e6-ff47-457e-8806-781259a6d687-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461108 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95pc\" (UniqueName: \"kubernetes.io/projected/6d9b74e6-ff47-457e-8806-781259a6d687-kube-api-access-m95pc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461121 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ww5p\" (UniqueName: \"kubernetes.io/projected/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-kube-api-access-9ww5p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461137 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a42248da-ec4d-4810-b331-250069ca99c8-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461150 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461163 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9b74e6-ff47-457e-8806-781259a6d687-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461176 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a42248da-ec4d-4810-b331-250069ca99c8-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461189 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5ab24eb-f971-4ff2-93c9-8a6539ed40db-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.461264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.461201 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2xh\" (UniqueName: \"kubernetes.io/projected/a42248da-ec4d-4810-b331-250069ca99c8-kube-api-access-pk2xh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:21:08.686964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.686923 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:21:08.699172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.699128 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-944ln"] Nov 03 17:21:08.736823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.736784 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:21:08.743718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.743679 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-twb8c"] Nov 03 17:21:08.780366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.779968 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:21:08.797279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.797239 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" path="/var/lib/kubelet/pods/6d9b74e6-ff47-457e-8806-781259a6d687/volumes" Nov 03 17:21:08.798231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.798204 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" path="/var/lib/kubelet/pods/d5ab24eb-f971-4ff2-93c9-8a6539ed40db/volumes" Nov 03 17:21:08.798805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:08.798781 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-gd4ql"] Nov 03 17:21:09.065643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.065607 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:21:09.067082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067048 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" Nov 03 17:21:09.067082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067076 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067125 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067133 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067146 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067154 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067180 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067188 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067239 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" Nov 03 17:21:09.067315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067277 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" Nov 03 17:21:09.067780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067517 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ab24eb-f971-4ff2-93c9-8a6539ed40db" containerName="istio-proxy" Nov 03 17:21:09.067780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067535 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1336b33-4119-41cf-b0bd-6458d1ead764" containerName="istio-proxy" Nov 03 17:21:09.067780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067548 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8678db30-57d9-402a-9b9e-f05362e0ef2a" containerName="istio-proxy" Nov 03 17:21:09.067780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067559 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9b74e6-ff47-457e-8806-781259a6d687" containerName="istio-proxy" Nov 03 17:21:09.067780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.067579 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a42248da-ec4d-4810-b331-250069ca99c8" containerName="istio-proxy" Nov 03 17:21:09.106546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.106510 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:21:09.106754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.106642 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.173835 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.173964 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174040 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174090 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174134 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn86s\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174221 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174589 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.175190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.174645 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276314 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276355 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276421 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276482 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276540 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276578 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276607 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.276950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.276917 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277000 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277000 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277058 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277222 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277391 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn86s\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.277564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.277493 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.279725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.279695 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.280179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.280153 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.302500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.302428 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.302675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.302650 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn86s\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:09.446997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:09.446895 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:10.261526 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:21:10.257183 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc5fad7_2039_4555_b8c0_eba537123b4f.slice/crio-e418885ebe3771adb34e885117fbaf5415fe2b67887f0bc56bb68f24cfe1db62 WatchSource:0}: Error finding container e418885ebe3771adb34e885117fbaf5415fe2b67887f0bc56bb68f24cfe1db62: Status 404 returned error can't find the container with id e418885ebe3771adb34e885117fbaf5415fe2b67887f0bc56bb68f24cfe1db62 Nov 03 17:21:10.261526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.258005 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:21:10.261526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.260998 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:10.261526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.261070 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:10.261526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.261101 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:10.371013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.370975 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" event={"ID":"edc5fad7-2039-4555-b8c0-eba537123b4f","Type":"ContainerStarted","Data":"e418885ebe3771adb34e885117fbaf5415fe2b67887f0bc56bb68f24cfe1db62"} Nov 03 17:21:10.797590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:10.797550 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a42248da-ec4d-4810-b331-250069ca99c8" path="/var/lib/kubelet/pods/a42248da-ec4d-4810-b331-250069ca99c8/volumes" Nov 03 17:21:11.387175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:11.387119 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" event={"ID":"edc5fad7-2039-4555-b8c0-eba537123b4f","Type":"ContainerStarted","Data":"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f"} Nov 03 17:21:11.454782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:11.450964 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:11.465747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:11.465708 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:11.466050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:11.466017 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:11.483217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:11.483137 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podStartSLOduration=2.483113588 podStartE2EDuration="2.483113588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:11.467251755 +0000 UTC m=+3966.800465336" watchObservedRunningTime="2026-11-03 17:21:11.483113588 +0000 UTC m=+3966.816327158" Nov 03 17:21:12.468708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.463936 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:12.468708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.464006 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:12.502748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.474606 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:21:12.502748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.474662 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:21:12.716484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.714317 6001 scope.go:122] "RemoveContainer" containerID="0e3c8674f2590df97bfc9f30bd8b4ed4a110c440afa5dd036a69e0754abf1e77" Nov 03 17:21:12.740244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.740213 6001 scope.go:122] "RemoveContainer" containerID="ba6ec013a496554bcdbc4d632a881716d8df45dc2ba20f0fb1f7d059c2e23ed0" Nov 03 17:21:12.782721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.782694 6001 scope.go:122] "RemoveContainer" containerID="883694032f8232f5d9fbb3e1bfa48550ab4c1d8bd9872defb6eeb34d1e983576" Nov 03 17:21:12.797576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:12.797552 6001 scope.go:122] "RemoveContainer" containerID="2aa43d3a30e7f21b09008bb6884b0ed100a6568b7916387034341f4519fc722f" Nov 03 17:21:13.407963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.407932 6001 generic.go:356] "Generic (PLEG): container finished" podID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerID="fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e" exitCode=0 Nov 03 17:21:13.408162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.408014 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerDied","Data":"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e"} Nov 03 17:21:13.408574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.408555 6001 scope.go:122] "RemoveContainer" containerID="fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e" Nov 03 17:21:13.410596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.410566 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:13.410705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.410648 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:13.410705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.410685 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:21:13.470745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.467835 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:13.470745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:13.467896 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:14.436985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:14.436947 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerStarted","Data":"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101"} Nov 03 17:21:14.460470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:14.459874 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:14.460470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:14.459956 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:15.373468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:15.372515 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:21:15.397205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:15.383795 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:15.397205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:15.383877 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:15.455470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:15.448127 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:15.455470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:15.448191 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:16.374114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:16.373887 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:16.374823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:16.374778 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:16.452551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:16.447838 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" start-of-body= Nov 03 17:21:16.452551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:16.447915 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.112:15021/healthz/ready\": dial tcp 10.134.1.112:15021: connect: connection refused" Nov 03 17:21:17.381871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:17.381832 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:17.382571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:17.382495 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:17.466971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:17.464813 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:17.466971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:17.466771 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:17.471840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:17.471807 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:21:18.370130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:18.370085 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:18.370343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:18.370149 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:19.370074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:19.370025 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:19.370672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:19.370106 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:20.370610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:20.370566 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" start-of-body= Nov 03 17:21:20.371061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:20.370634 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.107:15021/healthz/ready\": dial tcp 10.134.1.107:15021: connect: connection refused" Nov 03 17:21:21.375097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:21.375064 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:21:21.375638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:21.375580 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:21:21.376368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:21.376343 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:21:22.255438 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:22.255400 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:21:35.033022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:35.032981 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp"] Nov 03 17:21:35.045029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:35.044998 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c365dc46-xs7bp"] Nov 03 17:21:36.799923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:36.799869 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61785f8e-ad91-494a-ba1a-0744ce7ed33b" path="/var/lib/kubelet/pods/61785f8e-ad91-494a-ba1a-0744ce7ed33b/volumes" Nov 03 17:21:46.033672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:46.033630 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r"] Nov 03 17:21:46.046850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:46.046815 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0448f7a-z589r"] Nov 03 17:21:46.894468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:21:46.893771 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="579319fc-8242-4a15-98b0-bc0d3c23fb75" path="/var/lib/kubelet/pods/579319fc-8242-4a15-98b0-bc0d3c23fb75/volumes" Nov 03 17:21:52.253972 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:21:52.253932 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:22:11.567803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:11.567761 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 17:22:11.568268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:11.567982 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" containerID="cri-o://0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc" gracePeriod=30 Nov 03 17:22:13.730439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:13.730400 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 17:22:13.730941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:13.730656 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" containerID="cri-o://2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d" gracePeriod=30 Nov 03 17:22:15.334694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:15.334657 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:22:15.335192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:15.334732 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:22:16.948659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:16.948616 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 17:22:17.048658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048617 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.048882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048675 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.048882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048778 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.048882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048873 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048916 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.048934 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049082 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049130 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\") pod \"7266a271-f02f-45bd-befa-b68dc8e2826d\" (UID: \"7266a271-f02f-45bd-befa-b68dc8e2826d\") " Nov 03 17:22:17.049336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049293 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:17.049535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049512 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:17.049822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049802 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:17.049969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.049937 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:17.050205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.050179 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.050298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.050209 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.050298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.050224 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.050298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.050238 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7266a271-f02f-45bd-befa-b68dc8e2826d-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.050936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.050907 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:17.057302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.057278 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:17.057397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.057297 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:17.057627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.057607 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:17.058730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.058704 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q" pod "7266a271-f02f-45bd-befa-b68dc8e2826d" (UID: "7266a271-f02f-45bd-befa-b68dc8e2826d"). InnerVolumeSpecName "kube-api-access-j8g2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:17.152694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.152608 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.152694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.152644 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.152694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.152657 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.152694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.152672 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8g2q\" (UniqueName: \"kubernetes.io/projected/7266a271-f02f-45bd-befa-b68dc8e2826d-kube-api-access-j8g2q\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.152694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.152689 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7266a271-f02f-45bd-befa-b68dc8e2826d-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:17.248246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.248211 6001 generic.go:356] "Generic (PLEG): container finished" podID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerID="0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc" exitCode=0 Nov 03 17:22:17.248516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.248273 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" Nov 03 17:22:17.248516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.248298 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" event={"ID":"7266a271-f02f-45bd-befa-b68dc8e2826d","Type":"ContainerDied","Data":"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc"} Nov 03 17:22:17.248516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.248377 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z" event={"ID":"7266a271-f02f-45bd-befa-b68dc8e2826d","Type":"ContainerDied","Data":"9c1f9c6406e615ae69023af1228984b9b3a84b2bed09fcbe61b93c03993fba3a"} Nov 03 17:22:17.248516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.248405 6001 scope.go:122] "RemoveContainer" containerID="0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc" Nov 03 17:22:17.263515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.263438 6001 scope.go:122] "RemoveContainer" containerID="0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc" Nov 03 17:22:17.263914 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:17.263885 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc\": container with ID starting with 0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc not found: ID does not exist" containerID="0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc" Nov 03 17:22:17.264018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.263922 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc"} err="failed to get container status \"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc\": rpc error: code = NotFound desc = could not find container \"0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc\": container with ID starting with 0853c19dd8ee42ae475fcb3bf49ca8492d534037533fc0a24806c3477f01d7fc not found: ID does not exist" Nov 03 17:22:17.356208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.356161 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 17:22:17.389709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:17.389679 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-b527z"] Nov 03 17:22:18.021271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:18.021237 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:22:18.021696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:18.021317 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:22:18.795267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:18.795236 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" path="/var/lib/kubelet/pods/7266a271-f02f-45bd-befa-b68dc8e2826d/volumes" Nov 03 17:22:19.140792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.140763 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 17:22:19.205431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205389 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.205804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205611 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.205804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205655 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.205804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205717 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.205804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205773 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.206067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.205857 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.206067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.206022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.206236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.206219 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.207078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.207051 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.207190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.207097 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.207190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.207142 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.207190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.207175 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\") pod \"63cc4377-5e92-4fcb-abb0-e79d61146f86\" (UID: \"63cc4377-5e92-4fcb-abb0-e79d61146f86\") " Nov 03 17:22:19.207506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.207064 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:19.208716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.208693 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63cc4377-5e92-4fcb-abb0-e79d61146f86-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.208839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.208719 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.208839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.208732 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.208839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.208744 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.209424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.209334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:19.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.210662 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.210715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.210679 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:19.211229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.211202 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.219023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.218986 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b" pod "63cc4377-5e92-4fcb-abb0-e79d61146f86" (UID: "63cc4377-5e92-4fcb-abb0-e79d61146f86"). InnerVolumeSpecName "kube-api-access-tsj5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:19.270592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.270551 6001 generic.go:356] "Generic (PLEG): container finished" podID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerID="2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d" exitCode=0 Nov 03 17:22:19.270800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.270619 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" Nov 03 17:22:19.270800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.270644 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" event={"ID":"63cc4377-5e92-4fcb-abb0-e79d61146f86","Type":"ContainerDied","Data":"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d"} Nov 03 17:22:19.270800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.270703 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b" event={"ID":"63cc4377-5e92-4fcb-abb0-e79d61146f86","Type":"ContainerDied","Data":"0cc78963b08fdfcdee4d0dd43394ed8fc20d6cae424a671fd01e2814facc342a"} Nov 03 17:22:19.270800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.270729 6001 scope.go:122] "RemoveContainer" containerID="2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d" Nov 03 17:22:19.286333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.286311 6001 scope.go:122] "RemoveContainer" containerID="2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d" Nov 03 17:22:19.286714 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:19.286684 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d\": container with ID starting with 2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d not found: ID does not exist" containerID="2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d" Nov 03 17:22:19.286835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.286726 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d"} err="failed to get container status \"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d\": rpc error: code = NotFound desc = could not find container \"2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d\": container with ID starting with 2e44aebab8de21de3702d08816b08d99e2ae7ea2fb3b5e48754ddcd82495128d not found: ID does not exist" Nov 03 17:22:19.310075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.310044 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.310075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.310075 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.310244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.310088 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsj5b\" (UniqueName: \"kubernetes.io/projected/63cc4377-5e92-4fcb-abb0-e79d61146f86-kube-api-access-tsj5b\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.310244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.310101 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.310244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.310115 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63cc4377-5e92-4fcb-abb0-e79d61146f86-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.342815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.342776 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 17:22:19.347036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.347012 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3b6f35-openshift-default-7599f57489-zkq4b"] Nov 03 17:22:19.622040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.622006 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:22:19.622258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:19.622232 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" containerID="cri-o://2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384" gracePeriod=30 Nov 03 17:22:20.793040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:20.793004 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" path="/var/lib/kubelet/pods/63cc4377-5e92-4fcb-abb0-e79d61146f86/volumes" Nov 03 17:22:22.278780 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:22.278738 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:22:24.931563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:24.931535 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:22:25.022040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022008 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 17:22:25.022248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022223 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" containerID="cri-o://96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169" gracePeriod=30 Nov 03 17:22:25.022614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022585 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.022786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022683 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.022786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022774 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.022926 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023013 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:25.023073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023038 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:25.023172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023106 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023149 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023169 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023223 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023258 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\") pod \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\" (UID: \"b17eaa40-3d12-4f36-bdee-e4c44f657e11\") " Nov 03 17:22:25.023753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.023397 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:25.024923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.024892 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:25.026453 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.026415 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.026562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.026458 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.026562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.026473 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.026562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.026488 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.027738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.027424 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "kube-api-access-24k8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:25.027945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.027766 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:25.027945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.027912 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:25.028903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.028869 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:25.029602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.029578 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy" pod "b17eaa40-3d12-4f36-bdee-e4c44f657e11" (UID: "b17eaa40-3d12-4f36-bdee-e4c44f657e11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:25.137619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.129602 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.137619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.129641 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b17eaa40-3d12-4f36-bdee-e4c44f657e11-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.137619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.129656 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24k8w\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-kube-api-access-24k8w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.137619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.129671 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.137619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.129686 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b17eaa40-3d12-4f36-bdee-e4c44f657e11-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:25.327819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.327787 6001 generic.go:356] "Generic (PLEG): container finished" podID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerID="2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384" exitCode=0 Nov 03 17:22:25.328543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.327840 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" Nov 03 17:22:25.328543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.327872 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" event={"ID":"b17eaa40-3d12-4f36-bdee-e4c44f657e11","Type":"ContainerDied","Data":"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384"} Nov 03 17:22:25.328543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.327926 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz" event={"ID":"b17eaa40-3d12-4f36-bdee-e4c44f657e11","Type":"ContainerDied","Data":"1dc8f136bcafba2086d504dd85c933d51479a8e00a9a449b74966820092c1c11"} Nov 03 17:22:25.328543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.327951 6001 scope.go:122] "RemoveContainer" containerID="2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384" Nov 03 17:22:25.341402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.341384 6001 scope.go:122] "RemoveContainer" containerID="2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384" Nov 03 17:22:25.341720 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:25.341700 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384\": container with ID starting with 2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384 not found: ID does not exist" containerID="2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384" Nov 03 17:22:25.341792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.341728 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384"} err="failed to get container status \"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384\": rpc error: code = NotFound desc = could not find container \"2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384\": container with ID starting with 2676300f9c3ceab6cb2f626c6f1007d3a252f26e56f9d9cdf77941318669b384 not found: ID does not exist" Nov 03 17:22:25.430199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.430156 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:22:25.442152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:25.442110 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-6046c2-openshift-default-66c4774cd9-w6tqz"] Nov 03 17:22:26.792924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:26.792881 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" path="/var/lib/kubelet/pods/b17eaa40-3d12-4f36-bdee-e4c44f657e11/volumes" Nov 03 17:22:29.192927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:29.192882 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 17:22:29.193360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:29.193127 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" containerID="cri-o://340a2683f45610236fc49654a1d23a7a782a2c82229d04fb0391409adaa3a4a6" gracePeriod=30 Nov 03 17:22:30.385257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.385227 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 17:22:30.391746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.391711 6001 generic.go:356] "Generic (PLEG): container finished" podID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerID="96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169" exitCode=0 Nov 03 17:22:30.391938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.391910 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" Nov 03 17:22:30.391938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.391927 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" event={"ID":"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3","Type":"ContainerDied","Data":"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169"} Nov 03 17:22:30.392118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.391992 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd" event={"ID":"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3","Type":"ContainerDied","Data":"8ad7169f9f7c165c5168c55cc3c29b56d08322a42528dce2d02b6cec0c8ee494"} Nov 03 17:22:30.392118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.392016 6001 scope.go:122] "RemoveContainer" containerID="96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169" Nov 03 17:22:30.404134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.404118 6001 scope.go:122] "RemoveContainer" containerID="96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169" Nov 03 17:22:30.404391 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:30.404373 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169\": container with ID starting with 96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169 not found: ID does not exist" containerID="96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169" Nov 03 17:22:30.404436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.404399 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169"} err="failed to get container status \"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169\": rpc error: code = NotFound desc = could not find container \"96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169\": container with ID starting with 96c94a9fed584bc7199fa5222b78d1f712f37f72b054e74f67b5fe6b216a2169 not found: ID does not exist" Nov 03 17:22:30.470277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470244 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470294 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470340 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470431 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470691 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:30.470805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470724 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470774 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470839 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.470935 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470908 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:30.471055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470962 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.471055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.470999 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\") pod \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\" (UID: \"e0b166fa-1f1d-4e0e-95d2-bb2953244fe3\") " Nov 03 17:22:30.471890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.471370 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:30.471890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.471856 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:30.472103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.472058 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.472103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.472080 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.472325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.472095 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.472325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.472299 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.472803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.472766 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:30.474424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.474364 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:30.474677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.474530 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:30.474751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.474708 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "kube-api-access-2gkpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:30.474751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.474721 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy" pod "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" (UID: "e0b166fa-1f1d-4e0e-95d2-bb2953244fe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:30.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.573378 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.573408 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.573417 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.573425 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.573472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.573433 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkpc\" (UniqueName: \"kubernetes.io/projected/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3-kube-api-access-2gkpc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:30.742645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.742600 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 17:22:30.760389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.760346 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-4jdxd"] Nov 03 17:22:30.792075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:30.792045 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" path="/var/lib/kubelet/pods/e0b166fa-1f1d-4e0e-95d2-bb2953244fe3/volumes" Nov 03 17:22:34.439260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.439223 6001 generic.go:356] "Generic (PLEG): container finished" podID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerID="340a2683f45610236fc49654a1d23a7a782a2c82229d04fb0391409adaa3a4a6" exitCode=0 Nov 03 17:22:34.439916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.439296 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerDied","Data":"340a2683f45610236fc49654a1d23a7a782a2c82229d04fb0391409adaa3a4a6"} Nov 03 17:22:34.440064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.439974 6001 scope.go:122] "RemoveContainer" containerID="9050790ed792e206c05e678d31600a34e7312d57810bc9906548682f6681f029" Nov 03 17:22:34.513235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.513207 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 17:22:34.649143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649105 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649213 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649413 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649464 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:34.649508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649476 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649699 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.649834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649798 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:34.650079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.650198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.649930 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:34.650283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.650037 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:34.650283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.650185 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.650283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.650266 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\") pod \"b827585b-ffe1-4285-9a2f-7512655ebfdb\" (UID: \"b827585b-ffe1-4285-9a2f-7512655ebfdb\") " Nov 03 17:22:34.650566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.650541 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:34.651210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651185 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b827585b-ffe1-4285-9a2f-7512655ebfdb-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.651347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651216 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.651347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651231 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.651347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651247 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.651347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651263 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.651886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.651863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:34.652182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.652154 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:34.652328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.652308 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "kube-api-access-n82jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:34.652528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.652502 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token" pod "b827585b-ffe1-4285-9a2f-7512655ebfdb" (UID: "b827585b-ffe1-4285-9a2f-7512655ebfdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:34.752854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.752816 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.752854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.752847 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.752854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.752858 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82jw\" (UniqueName: \"kubernetes.io/projected/b827585b-ffe1-4285-9a2f-7512655ebfdb-kube-api-access-n82jw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:34.753118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:34.752868 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b827585b-ffe1-4285-9a2f-7512655ebfdb-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:35.454197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.454132 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" event={"ID":"b827585b-ffe1-4285-9a2f-7512655ebfdb","Type":"ContainerDied","Data":"486ce76ff31c7a5ee6b8156c5020c8070561fe65fa066188c94daec8c636a9e9"} Nov 03 17:22:35.454783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.454266 6001 scope.go:122] "RemoveContainer" containerID="340a2683f45610236fc49654a1d23a7a782a2c82229d04fb0391409adaa3a4a6" Nov 03 17:22:35.454783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.454508 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2" Nov 03 17:22:35.514575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.514534 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 17:22:35.527638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.527602 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-jwkj2"] Nov 03 17:22:35.736923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.736875 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:22:35.737195 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:35.737168 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" containerID="cri-o://de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559" gracePeriod=30 Nov 03 17:22:36.821498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:36.811303 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" path="/var/lib/kubelet/pods/b827585b-ffe1-4285-9a2f-7512655ebfdb/volumes" Nov 03 17:22:41.040513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.040486 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:22:41.185191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185157 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185311 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185359 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185384 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185416 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:41.185538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185422 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185492 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185573 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185697 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:41.185795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185770 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185790 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:41.185907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185799 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:41.185907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185805 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\") pod \"1425396b-f582-4528-a8c9-0b2969cfb8c0\" (UID: \"1425396b-f582-4528-a8c9-0b2969cfb8c0\") " Nov 03 17:22:41.185907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.185837 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:41.186851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.186820 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1425396b-f582-4528-a8c9-0b2969cfb8c0-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.186851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.186852 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.187043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.186867 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.187043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.186882 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.187043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.186896 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.187900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.187875 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:41.188001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.187925 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:41.188001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.187966 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "kube-api-access-2kqf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:41.188148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.188125 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy" pod "1425396b-f582-4528-a8c9-0b2969cfb8c0" (UID: "1425396b-f582-4528-a8c9-0b2969cfb8c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:41.288842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.288760 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kqf6\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-kube-api-access-2kqf6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.288842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.288793 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.288842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.288803 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.288842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.288812 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1425396b-f582-4528-a8c9-0b2969cfb8c0-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:22:41.511577 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.511539 6001 generic.go:356] "Generic (PLEG): container finished" podID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerID="de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559" exitCode=0 Nov 03 17:22:41.511736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.511608 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" Nov 03 17:22:41.511736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.511615 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" event={"ID":"1425396b-f582-4528-a8c9-0b2969cfb8c0","Type":"ContainerDied","Data":"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559"} Nov 03 17:22:41.511736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.511696 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544" event={"ID":"1425396b-f582-4528-a8c9-0b2969cfb8c0","Type":"ContainerDied","Data":"75ab90e6127de380a319e57638c97fc70247e9d2991c1080e4a4d0c1f55623cc"} Nov 03 17:22:41.511736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.511720 6001 scope.go:122] "RemoveContainer" containerID="de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559" Nov 03 17:22:41.525060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.525043 6001 scope.go:122] "RemoveContainer" containerID="de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559" Nov 03 17:22:41.525337 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:41.525319 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559\": container with ID starting with de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559 not found: ID does not exist" containerID="de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559" Nov 03 17:22:41.525392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.525345 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559"} err="failed to get container status \"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559\": rpc error: code = NotFound desc = could not find container \"de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559\": container with ID starting with de1128c6e1b5a041d097c80af010868ffc592ecc8f76f400b4c43eb7568a9559 not found: ID does not exist" Nov 03 17:22:41.573112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.573017 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:22:41.619518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:41.616169 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-da409b-openshift-default-555bd4d749-m5544"] Nov 03 17:22:42.792115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:22:42.792081 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" path="/var/lib/kubelet/pods/1425396b-f582-4528-a8c9-0b2969cfb8c0/volumes" Nov 03 17:22:52.252107 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:22:52.252023 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:23:12.932209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:12.932176 6001 scope.go:122] "RemoveContainer" containerID="fd7c5a7703eae9065527011fb1e33095b7df0bda440f86eb6c1a2adce782deac" Nov 03 17:23:12.943860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:12.943838 6001 scope.go:122] "RemoveContainer" containerID="af82418287dfd0bce58c5a4143a9bb33553e06b09e43860e49d017180c1e5734" Nov 03 17:23:22.253764 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:23:22.253724 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:23:28.358208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.358167 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw"] Nov 03 17:23:28.359406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359380 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.359406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359407 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359476 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359486 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359535 6001 cpu_manager.go:414] "containerMap: removing container" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359544 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359563 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359570 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359609 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359616 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359629 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" Nov 03 17:23:28.359661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359636 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359893 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7266a271-f02f-45bd-befa-b68dc8e2826d" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359910 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17eaa40-3d12-4f36-bdee-e4c44f657e11" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359929 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359944 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b166fa-1f1d-4e0e-95d2-bb2953244fe3" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359958 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cc4377-5e92-4fcb-abb0-e79d61146f86" containerName="istio-proxy" Nov 03 17:23:28.360158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.359969 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1425396b-f582-4528-a8c9-0b2969cfb8c0" containerName="istio-proxy" Nov 03 17:23:28.365762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.365738 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.382429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.382403 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw"] Nov 03 17:23:28.389282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.389255 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:23:28.389503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.389363 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:23:28.389503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.389377 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:23:28.389503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.389409 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:23:28.418782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.418740 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwlm\" (UniqueName: \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.419068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.419048 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.419128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.419114 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.520571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.520524 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwlm\" (UniqueName: \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.520774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.520705 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.520774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.520743 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.521397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.521364 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.523200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.523174 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.545063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.545032 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwlm\" (UniqueName: \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\") pod \"maas-api-revoke-sub-aaf8e168-s9nbw\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.683226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.683122 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:28.859592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.859557 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt"] Nov 03 17:23:28.860586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.860570 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.860637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.860587 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.860746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.860736 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b827585b-ffe1-4285-9a2f-7512655ebfdb" containerName="istio-proxy" Nov 03 17:23:28.865971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.865955 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:28.883959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.883929 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt"] Nov 03 17:23:28.928502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.928427 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:28.928826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.928794 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:28.929103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.929077 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgmw\" (UniqueName: \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:28.989369 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:23:28.989340 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2dcf4f2_4cc0_4442_a52c_d94e17b8dad8.slice/crio-77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2 WatchSource:0}: Error finding container 77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2: Status 404 returned error can't find the container with id 77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2 Nov 03 17:23:28.990717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:28.990691 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw"] Nov 03 17:23:29.031183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.031150 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.031316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.031232 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.031425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.031408 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgmw\" (UniqueName: \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.031938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.031913 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.033836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.033814 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.051813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.051780 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" event={"ID":"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8","Type":"ContainerStarted","Data":"77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2"} Nov 03 17:23:29.056184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.056150 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgmw\" (UniqueName: \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\") pod \"maas-api-revoke-sub-bfe63895-lccmt\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.183405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.183371 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:29.502760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:29.502726 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt"] Nov 03 17:23:29.503168 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:23:29.502724 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebeaac82_b8c9_4639_9a59_1da50551fbc4.slice/crio-35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662 WatchSource:0}: Error finding container 35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662: Status 404 returned error can't find the container with id 35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662 Nov 03 17:23:30.068023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:30.067577 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" event={"ID":"ebeaac82-b8c9-4639-9a59-1da50551fbc4","Type":"ContainerStarted","Data":"b36b7839420975c9d266550d65f75eebe9bcc02605b182c2492af1cbdf4e275a"} Nov 03 17:23:30.068023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:30.067654 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" event={"ID":"ebeaac82-b8c9-4639-9a59-1da50551fbc4","Type":"ContainerStarted","Data":"35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662"} Nov 03 17:23:30.070264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:30.070222 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" event={"ID":"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8","Type":"ContainerStarted","Data":"6c67da8a771958bb4ccd0fcde91797aa2dabfb01e71e53115b7109334d52fab3"} Nov 03 17:23:30.114404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:30.114346 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" podStartSLOduration=2.114332921 podStartE2EDuration="2.114332921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:30.111215261 +0000 UTC m=+4105.444428831" watchObservedRunningTime="2026-11-03 17:23:30.114332921 +0000 UTC m=+4105.447546489" Nov 03 17:23:30.157513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:30.157430 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" podStartSLOduration=2.157416199 podStartE2EDuration="2.157416199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:30.154228098 +0000 UTC m=+4105.487441669" watchObservedRunningTime="2026-11-03 17:23:30.157416199 +0000 UTC m=+4105.490629768" Nov 03 17:23:31.082791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:31.082753 6001 generic.go:356] "Generic (PLEG): container finished" podID="ebeaac82-b8c9-4639-9a59-1da50551fbc4" containerID="b36b7839420975c9d266550d65f75eebe9bcc02605b182c2492af1cbdf4e275a" exitCode=0 Nov 03 17:23:31.083278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:31.082844 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" event={"ID":"ebeaac82-b8c9-4639-9a59-1da50551fbc4","Type":"ContainerDied","Data":"b36b7839420975c9d266550d65f75eebe9bcc02605b182c2492af1cbdf4e275a"} Nov 03 17:23:31.084657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:31.084625 6001 generic.go:356] "Generic (PLEG): container finished" podID="a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" containerID="6c67da8a771958bb4ccd0fcde91797aa2dabfb01e71e53115b7109334d52fab3" exitCode=0 Nov 03 17:23:31.084806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:31.084668 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" event={"ID":"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8","Type":"ContainerDied","Data":"6c67da8a771958bb4ccd0fcde91797aa2dabfb01e71e53115b7109334d52fab3"} Nov 03 17:23:32.297421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.297390 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:32.334307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.334279 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:32.409564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.409469 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\") pod \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " Nov 03 17:23:32.409742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.409723 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\" (UniqueName: \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\") pod \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " Nov 03 17:23:32.409812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.409802 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\") pod \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " Nov 03 17:23:32.409886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.409867 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\") pod \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\" (UID: \"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8\") " Nov 03 17:23:32.410247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.410217 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca" pod "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" (UID: "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:23:32.410247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.410228 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca" pod "ebeaac82-b8c9-4639-9a59-1da50551fbc4" (UID: "ebeaac82-b8c9-4639-9a59-1da50551fbc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:23:32.410494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.410335 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\") pod \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " Nov 03 17:23:32.410494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.410461 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\" (UniqueName: \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\") pod \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\" (UID: \"ebeaac82-b8c9-4639-9a59-1da50551fbc4\") " Nov 03 17:23:32.411623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.411591 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:32.411779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.411626 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:32.413047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.413011 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm" pod "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" (UID: "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8"). InnerVolumeSpecName "kube-api-access-5hwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:32.413193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.413085 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca" pod "ebeaac82-b8c9-4639-9a59-1da50551fbc4" (UID: "ebeaac82-b8c9-4639-9a59-1da50551fbc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:32.413193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.413177 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca" pod "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" (UID: "a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:32.413514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.413466 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw" pod "ebeaac82-b8c9-4639-9a59-1da50551fbc4" (UID: "ebeaac82-b8c9-4639-9a59-1da50551fbc4"). InnerVolumeSpecName "kube-api-access-tdgmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:32.513560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.513524 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:32.513560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.513555 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwlm\" (UniqueName: \"kubernetes.io/projected/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8-kube-api-access-5hwlm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:32.513560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.513567 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ebeaac82-b8c9-4639-9a59-1da50551fbc4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:32.513811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:32.513576 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgmw\" (UniqueName: \"kubernetes.io/projected/ebeaac82-b8c9-4639-9a59-1da50551fbc4-kube-api-access-tdgmw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:33.104019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.103922 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" Nov 03 17:23:33.104019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.103945 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw" event={"ID":"a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8","Type":"ContainerDied","Data":"77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2"} Nov 03 17:23:33.104241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.104034 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77819c10ff391aa871dbe8554daab8e606d8b2f407aac98ad76754986e3510f2" Nov 03 17:23:33.105919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.105891 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" event={"ID":"ebeaac82-b8c9-4639-9a59-1da50551fbc4","Type":"ContainerDied","Data":"35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662"} Nov 03 17:23:33.106069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.105928 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt" Nov 03 17:23:33.106069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:33.105941 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b426f8952b5cf6e3d17a6e4f44d8d36c9f0b3202289bc6abaf8e8632778662" Nov 03 17:23:39.678103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.678066 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw"] Nov 03 17:23:39.679279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679256 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" containerName="revoke-keys" Nov 03 17:23:39.679279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679280 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" containerName="revoke-keys" Nov 03 17:23:39.679422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679302 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ebeaac82-b8c9-4639-9a59-1da50551fbc4" containerName="revoke-keys" Nov 03 17:23:39.679422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679307 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebeaac82-b8c9-4639-9a59-1da50551fbc4" containerName="revoke-keys" Nov 03 17:23:39.679537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679492 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" containerName="revoke-keys" Nov 03 17:23:39.679537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.679504 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebeaac82-b8c9-4639-9a59-1da50551fbc4" containerName="revoke-keys" Nov 03 17:23:39.684486 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.684462 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.700110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.700079 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:23:39.700110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.700079 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:23:39.700341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.700129 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:23:39.700341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.700196 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:23:39.701500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.701426 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw"] Nov 03 17:23:39.782702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.782662 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.782954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.782928 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpcn\" (UniqueName: \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.783058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.783046 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.885124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.885080 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.885356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.885249 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpcn\" (UniqueName: \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.885356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.885310 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.885912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.885886 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.887954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.887934 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:39.942525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:39.941506 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpcn\" (UniqueName: \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\") pod \"maas-api-revoke-sub-e1539e8a-8k8cw\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:40.006468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.005179 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:40.345103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.345066 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8"] Nov 03 17:23:40.352070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.352042 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.369049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.369016 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8"] Nov 03 17:23:40.385094 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:23:40.385034 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7081157f_d26e_4812_8db2_94875fb35a24.slice/crio-4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a WatchSource:0}: Error finding container 4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a: Status 404 returned error can't find the container with id 4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a Nov 03 17:23:40.385750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.385724 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw"] Nov 03 17:23:40.399289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.399242 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.399437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.399330 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.399581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.399510 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxbfb\" (UniqueName: \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.501295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.501261 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.501505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.501318 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.501505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.501425 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbfb\" (UniqueName: \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.502156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.502133 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.544761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.544722 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.544942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.544895 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbfb\" (UniqueName: \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\") pod \"maas-api-revoke-sub-d576607e-m82q8\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.672383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.672351 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:40.984021 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:23:40.983984 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacc4860_3437_4210_beaa_592d39869526.slice/crio-792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7 WatchSource:0}: Error finding container 792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7: Status 404 returned error can't find the container with id 792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7 Nov 03 17:23:40.986043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:40.986002 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8"] Nov 03 17:23:41.232967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:41.232811 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" event={"ID":"7081157f-d26e-4812-8db2-94875fb35a24","Type":"ContainerStarted","Data":"d54278b4dceea411e01bda407fb7a7a65a7e14240552c703705725ca94f9670f"} Nov 03 17:23:41.232967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:41.232874 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" event={"ID":"7081157f-d26e-4812-8db2-94875fb35a24","Type":"ContainerStarted","Data":"4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a"} Nov 03 17:23:41.234753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:41.234726 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" event={"ID":"cacc4860-3437-4210-beaa-592d39869526","Type":"ContainerStarted","Data":"792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7"} Nov 03 17:23:41.278925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:41.278867 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" podStartSLOduration=2.278847912 podStartE2EDuration="2.278847912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:23:41.275826611 +0000 UTC m=+4116.609040180" watchObservedRunningTime="2026-11-03 17:23:41.278847912 +0000 UTC m=+4116.612061481" Nov 03 17:23:42.246480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:42.246428 6001 generic.go:356] "Generic (PLEG): container finished" podID="7081157f-d26e-4812-8db2-94875fb35a24" containerID="d54278b4dceea411e01bda407fb7a7a65a7e14240552c703705725ca94f9670f" exitCode=0 Nov 03 17:23:42.246958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:42.246527 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" event={"ID":"7081157f-d26e-4812-8db2-94875fb35a24","Type":"ContainerDied","Data":"d54278b4dceea411e01bda407fb7a7a65a7e14240552c703705725ca94f9670f"} Nov 03 17:23:42.248388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:42.248358 6001 generic.go:356] "Generic (PLEG): container finished" podID="cacc4860-3437-4210-beaa-592d39869526" containerID="f9737921c87eeaee33e7878a7d5e2e21ec178c755f4a0cf7909c0867f2c9d6a0" exitCode=0 Nov 03 17:23:42.248565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:42.248406 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" event={"ID":"cacc4860-3437-4210-beaa-592d39869526","Type":"ContainerDied","Data":"f9737921c87eeaee33e7878a7d5e2e21ec178c755f4a0cf7909c0867f2c9d6a0"} Nov 03 17:23:43.482912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.482877 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:43.527017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.526931 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:43.574007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.573968 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\") pod \"7081157f-d26e-4812-8db2-94875fb35a24\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " Nov 03 17:23:43.574390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.574348 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\") pod \"cacc4860-3437-4210-beaa-592d39869526\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " Nov 03 17:23:43.574619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.574542 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\" (UniqueName: \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\") pod \"cacc4860-3437-4210-beaa-592d39869526\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " Nov 03 17:23:43.574742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.574655 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\") pod \"7081157f-d26e-4812-8db2-94875fb35a24\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " Nov 03 17:23:43.574824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.574756 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\") pod \"cacc4860-3437-4210-beaa-592d39869526\" (UID: \"cacc4860-3437-4210-beaa-592d39869526\") " Nov 03 17:23:43.574886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.574871 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\" (UniqueName: \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\") pod \"7081157f-d26e-4812-8db2-94875fb35a24\" (UID: \"7081157f-d26e-4812-8db2-94875fb35a24\") " Nov 03 17:23:43.577105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.577075 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca" pod "cacc4860-3437-4210-beaa-592d39869526" (UID: "cacc4860-3437-4210-beaa-592d39869526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:23:43.577262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.577099 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca" pod "7081157f-d26e-4812-8db2-94875fb35a24" (UID: "7081157f-d26e-4812-8db2-94875fb35a24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:43.577364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.577313 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca" pod "7081157f-d26e-4812-8db2-94875fb35a24" (UID: "7081157f-d26e-4812-8db2-94875fb35a24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:23:43.579873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.579843 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca" pod "cacc4860-3437-4210-beaa-592d39869526" (UID: "cacc4860-3437-4210-beaa-592d39869526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:23:43.581437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.581402 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn" pod "7081157f-d26e-4812-8db2-94875fb35a24" (UID: "7081157f-d26e-4812-8db2-94875fb35a24"). InnerVolumeSpecName "kube-api-access-wmpcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:43.581961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.581937 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb" pod "cacc4860-3437-4210-beaa-592d39869526" (UID: "cacc4860-3437-4210-beaa-592d39869526"). InnerVolumeSpecName "kube-api-access-xxbfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:23:43.678637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678587 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cacc4860-3437-4210-beaa-592d39869526-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:43.678637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678633 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpcn\" (UniqueName: \"kubernetes.io/projected/7081157f-d26e-4812-8db2-94875fb35a24-kube-api-access-wmpcn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:43.678637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678644 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:43.678893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678653 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cacc4860-3437-4210-beaa-592d39869526-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:43.678893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678662 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxbfb\" (UniqueName: \"kubernetes.io/projected/cacc4860-3437-4210-beaa-592d39869526-kube-api-access-xxbfb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:43.678893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:43.678672 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7081157f-d26e-4812-8db2-94875fb35a24-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:23:44.269163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.269128 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" Nov 03 17:23:44.269163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.269136 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw" event={"ID":"7081157f-d26e-4812-8db2-94875fb35a24","Type":"ContainerDied","Data":"4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a"} Nov 03 17:23:44.269428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.269193 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a73765d40f0736c698277a707abde496c10ee40737ef7f6b2313105527caa9a" Nov 03 17:23:44.270782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.270763 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" Nov 03 17:23:44.270944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.270803 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8" event={"ID":"cacc4860-3437-4210-beaa-592d39869526","Type":"ContainerDied","Data":"792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7"} Nov 03 17:23:44.270944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:23:44.270850 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="792b98ab73e984912e0ddc7b4bba2841923e702152440541726372f609898dc7" Nov 03 17:23:52.254060 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:23:52.254017 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:24:10.250683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:10.250639 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:24:15.251322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:15.251268 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" containerID="cri-o://f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2" gracePeriod=25 Nov 03 17:24:22.254178 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:24:22.254089 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:24:22.850423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:22.850381 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:32.849320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:32.849275 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:35.547274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.547243 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:24:35.604630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.604596 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\" (UniqueName: \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\") pod \"e97485b6-2f22-4247-9c43-992406601b59\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " Nov 03 17:24:35.604909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.604891 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\") pod \"e97485b6-2f22-4247-9c43-992406601b59\" (UID: \"e97485b6-2f22-4247-9c43-992406601b59\") " Nov 03 17:24:35.605269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.605245 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume" pod "e97485b6-2f22-4247-9c43-992406601b59" (UID: "e97485b6-2f22-4247-9c43-992406601b59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:35.605642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.605623 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e97485b6-2f22-4247-9c43-992406601b59-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:35.606932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.606905 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm" pod "e97485b6-2f22-4247-9c43-992406601b59" (UID: "e97485b6-2f22-4247-9c43-992406601b59"). InnerVolumeSpecName "kube-api-access-z9djm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:35.684531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.684489 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:24:35.684793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.684765 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" containerID="cri-o://e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a" gracePeriod=30 Nov 03 17:24:35.707507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:35.707466 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9djm\" (UniqueName: \"kubernetes.io/projected/e97485b6-2f22-4247-9c43-992406601b59-kube-api-access-z9djm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:36.026690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.026469 6001 generic.go:356] "Generic (PLEG): container finished" podID="e97485b6-2f22-4247-9c43-992406601b59" containerID="f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2" exitCode=0 Nov 03 17:24:36.027022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.026979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" event={"ID":"e97485b6-2f22-4247-9c43-992406601b59","Type":"ContainerDied","Data":"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2"} Nov 03 17:24:36.027169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.027153 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" event={"ID":"e97485b6-2f22-4247-9c43-992406601b59","Type":"ContainerDied","Data":"d7ba64fdaaed51973f41b9cd68d564ee25086928a955200f38b4ffb906ecd695"} Nov 03 17:24:36.027278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.027266 6001 scope.go:122] "RemoveContainer" containerID="f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2" Nov 03 17:24:36.027549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.027531 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5" Nov 03 17:24:36.076681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.075620 6001 scope.go:122] "RemoveContainer" containerID="f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2" Nov 03 17:24:36.081586 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:24:36.081511 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2\": container with ID starting with f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2 not found: ID does not exist" containerID="f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2" Nov 03 17:24:36.081841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.081800 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2"} err="failed to get container status \"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2\": rpc error: code = NotFound desc = could not find container \"f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2\": container with ID starting with f28040e6b92ce5b4b66d5146cca2311f8e012111fa4038df05807d0874b11ba2 not found: ID does not exist" Nov 03 17:24:36.111088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.111052 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:24:36.115924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.115893 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-24cb6d-8cb94b55f-t54z5"] Nov 03 17:24:36.812913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:36.812868 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97485b6-2f22-4247-9c43-992406601b59" path="/var/lib/kubelet/pods/e97485b6-2f22-4247-9c43-992406601b59/volumes" Nov 03 17:24:39.203049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:39.203012 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:24:39.203514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:39.203245 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" containerID="cri-o://8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3" gracePeriod=30 Nov 03 17:24:40.997892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:40.997865 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:24:41.057199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.057154 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.057393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.057251 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.057730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.057693 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.057871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.057848 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.057986 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058095 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058111 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.058291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058261 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058300 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058378 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058418 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\") pod \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\" (UID: \"0818e36f-3d2a-4ab2-b48e-311247ba15fc\") " Nov 03 17:24:41.058503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.058488 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:41.059603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.059577 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.059731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.059607 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.059731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.059626 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.060037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.059946 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.060037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.059962 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.061826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.061552 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "kube-api-access-7lp2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:41.063646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.063620 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:41.063769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.063700 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.069180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.069148 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token" pod "0818e36f-3d2a-4ab2-b48e-311247ba15fc" (UID: "0818e36f-3d2a-4ab2-b48e-311247ba15fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:41.084290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.084254 6001 generic.go:356] "Generic (PLEG): container finished" podID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerID="e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a" exitCode=0 Nov 03 17:24:41.084487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.084317 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" Nov 03 17:24:41.084661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.084636 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" event={"ID":"0818e36f-3d2a-4ab2-b48e-311247ba15fc","Type":"ContainerDied","Data":"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a"} Nov 03 17:24:41.084762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.084743 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx" event={"ID":"0818e36f-3d2a-4ab2-b48e-311247ba15fc","Type":"ContainerDied","Data":"b5aa43a5934e8f17ba16d5ece285d281ac85523a3ded8b5b3827cd81cb967313"} Nov 03 17:24:41.084832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.084785 6001 scope.go:122] "RemoveContainer" containerID="e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a" Nov 03 17:24:41.142747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.142718 6001 scope.go:122] "RemoveContainer" containerID="e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a" Nov 03 17:24:41.143343 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:24:41.143182 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a\": container with ID starting with e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a not found: ID does not exist" containerID="e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a" Nov 03 17:24:41.143343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.143224 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a"} err="failed to get container status \"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a\": rpc error: code = NotFound desc = could not find container \"e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a\": container with ID starting with e56478ae19e3d0e5e0fda6434a552179070cd9009056433e653b430162e0162a not found: ID does not exist" Nov 03 17:24:41.149998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.149961 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:24:41.155484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.155414 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-r6kqx"] Nov 03 17:24:41.162283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162249 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lp2j\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-kube-api-access-7lp2j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.162409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162285 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.162409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162301 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.162409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162314 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.162409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162331 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.162409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:41.162344 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0818e36f-3d2a-4ab2-b48e-311247ba15fc-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:42.799079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:42.799046 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" path="/var/lib/kubelet/pods/0818e36f-3d2a-4ab2-b48e-311247ba15fc/volumes" Nov 03 17:24:43.324558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.324516 6001 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:24:43.324737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.324575 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:24:43.384784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.384748 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:24:43.385759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385742 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7081157f-d26e-4812-8db2-94875fb35a24" containerName="revoke-keys" Nov 03 17:24:43.385820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385760 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7081157f-d26e-4812-8db2-94875fb35a24" containerName="revoke-keys" Nov 03 17:24:43.385820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385784 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cacc4860-3437-4210-beaa-592d39869526" containerName="revoke-keys" Nov 03 17:24:43.385820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385790 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cacc4860-3437-4210-beaa-592d39869526" containerName="revoke-keys" Nov 03 17:24:43.385928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385849 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" Nov 03 17:24:43.385928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385854 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" Nov 03 17:24:43.385928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385865 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" Nov 03 17:24:43.385928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.385873 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" Nov 03 17:24:43.386515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.386025 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cacc4860-3437-4210-beaa-592d39869526" containerName="revoke-keys" Nov 03 17:24:43.386515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.386043 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0818e36f-3d2a-4ab2-b48e-311247ba15fc" containerName="istio-proxy" Nov 03 17:24:43.386515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.386052 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97485b6-2f22-4247-9c43-992406601b59" containerName="payload-processing" Nov 03 17:24:43.386515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.386062 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7081157f-d26e-4812-8db2-94875fb35a24" containerName="revoke-keys" Nov 03 17:24:43.390185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.390167 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.410199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.410165 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-547925-openshift-default-dockercfg-5rrq9\"" Nov 03 17:24:43.441800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.426619 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:24:43.536696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.536651 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm5f8\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.536920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.536707 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.536920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.536866 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.536986 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.537024 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.537050 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.537099 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.537178 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.537250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.537197 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.638745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638641 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm5f8\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.638745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638685 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.638745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638742 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638815 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638849 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638877 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.638944 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639048 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639082 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639298 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639348 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639407 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639420 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.639663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.639642 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.641400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.641377 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.641550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.641527 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.663194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.663161 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.663439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.663299 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm5f8\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\") pod \"e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.706321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.706284 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:43.824115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:43.824077 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:24:44.038736 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:24:44.038705 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd79931_579e_4fb5_8a58_6819302d32cb.slice/crio-4d5ae7ba0c09e8f860b346bd03075dca6a4fd81c7510cf01168136ba5817433f WatchSource:0}: Error finding container 4d5ae7ba0c09e8f860b346bd03075dca6a4fd81c7510cf01168136ba5817433f: Status 404 returned error can't find the container with id 4d5ae7ba0c09e8f860b346bd03075dca6a4fd81c7510cf01168136ba5817433f Nov 03 17:24:44.038851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.038784 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:24:44.042187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.042148 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:24:44.042340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.042241 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:24:44.042340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.042291 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:24:44.167185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.167139 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" event={"ID":"bdd79931-579e-4fb5-8a58-6819302d32cb","Type":"ContainerStarted","Data":"4d5ae7ba0c09e8f860b346bd03075dca6a4fd81c7510cf01168136ba5817433f"} Nov 03 17:24:44.837353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:44.837315 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:24:45.003746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.003711 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.003937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.003766 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.003937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.003806 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.003937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.003909 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.003970 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004070 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004162 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:45.004252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004213 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004235 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:45.004428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004283 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004364 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\") pod \"64399d01-9590-4dee-a4ac-5077a4acfc70\" (UID: \"64399d01-9590-4dee-a4ac-5077a4acfc70\") " Nov 03 17:24:45.004570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004456 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:45.004719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004690 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:45.005030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.004992 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:45.005337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.005310 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.005429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.005344 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64399d01-9590-4dee-a4ac-5077a4acfc70-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.005429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.005358 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.005429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.005370 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.005429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.005385 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.006668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.006638 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:45.007076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.007052 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:45.007508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.007433 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "kube-api-access-cl588". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:45.007780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.007762 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token" pod "64399d01-9590-4dee-a4ac-5077a4acfc70" (UID: "64399d01-9590-4dee-a4ac-5077a4acfc70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:45.106576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.106544 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.106576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.106571 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.106576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.106583 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl588\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-kube-api-access-cl588\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.106854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.106593 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64399d01-9590-4dee-a4ac-5077a4acfc70-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:45.185353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.185026 6001 generic.go:356] "Generic (PLEG): container finished" podID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerID="8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3" exitCode=0 Nov 03 17:24:45.185633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.185270 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" Nov 03 17:24:45.198997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.185316 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" event={"ID":"64399d01-9590-4dee-a4ac-5077a4acfc70","Type":"ContainerDied","Data":"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3"} Nov 03 17:24:45.198997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.198607 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g" event={"ID":"64399d01-9590-4dee-a4ac-5077a4acfc70","Type":"ContainerDied","Data":"cf05fa4c3df4b36c5186042c801fd8f22afabf455e17fc208469a64eddef33c6"} Nov 03 17:24:45.198997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.198670 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" event={"ID":"bdd79931-579e-4fb5-8a58-6819302d32cb","Type":"ContainerStarted","Data":"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e"} Nov 03 17:24:45.198997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.198718 6001 scope.go:122] "RemoveContainer" containerID="8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3" Nov 03 17:24:45.215011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.214989 6001 scope.go:122] "RemoveContainer" containerID="8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3" Nov 03 17:24:45.215377 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:24:45.215348 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3\": container with ID starting with 8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3 not found: ID does not exist" containerID="8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3" Nov 03 17:24:45.215545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.215388 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3"} err="failed to get container status \"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3\": rpc error: code = NotFound desc = could not find container \"8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3\": container with ID starting with 8d5e6bf6f339067cce0a7a47b353e9e062b973456ade225b598fd62526d34dd3 not found: ID does not exist" Nov 03 17:24:45.255017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.254886 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podStartSLOduration=2.254862916 podStartE2EDuration="2.254862916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:45.248457663 +0000 UTC m=+4180.581671225" watchObservedRunningTime="2026-11-03 17:24:45.254862916 +0000 UTC m=+4180.588076486" Nov 03 17:24:45.307106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.307067 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:24:45.314365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.314327 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-vb48g"] Nov 03 17:24:45.707280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.707243 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:45.709025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.708991 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:45.709180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.709047 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:45.738826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.738792 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m"] Nov 03 17:24:45.739805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.739774 6001 cpu_manager.go:414] "containerMap: removing container" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" Nov 03 17:24:45.739805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.739807 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" Nov 03 17:24:45.739998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.739988 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" containerName="istio-proxy" Nov 03 17:24:45.771492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.764729 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:45.777975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.777938 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m"] Nov 03 17:24:45.790301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.790267 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:24:45.790971 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.790942 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:24:45.791108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.791094 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:24:45.791185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.791165 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:24:45.925629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.925539 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pph9d\" (UniqueName: \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:45.926135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.926109 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:45.926348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:45.926329 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.027733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.027698 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.027934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.027819 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.027934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.027871 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pph9d\" (UniqueName: \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.028364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.028340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.030272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.030246 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.072710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.072610 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pph9d\" (UniqueName: \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\") pod \"maas-api-revoke-sub-082500be-mr29m\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.099087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.099058 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:46.434174 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:24:46.434103 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366af900_f825_402b_b013_b93794ec3c68.slice/crio-f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a WatchSource:0}: Error finding container f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a: Status 404 returned error can't find the container with id f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a Nov 03 17:24:46.434174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.434125 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m"] Nov 03 17:24:46.707298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.707261 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:46.707513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.707312 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:46.803684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.803650 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64399d01-9590-4dee-a4ac-5077a4acfc70" path="/var/lib/kubelet/pods/64399d01-9590-4dee-a4ac-5077a4acfc70/volumes" Nov 03 17:24:46.812439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.812410 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7"] Nov 03 17:24:46.819154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.819128 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.836848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.836816 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7"] Nov 03 17:24:46.844461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.842715 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.844461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.842945 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.844461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.843122 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.969498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.958694 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.970256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.970218 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.970650 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.970573 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.971744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.971719 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.971865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.971827 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:46.994949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:46.994918 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\") pod \"maas-api-revoke-sub-48545c07-nwmh7\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:47.136960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.136920 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:47.233579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.233536 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" event={"ID":"366af900-f825-402b-b013-b93794ec3c68","Type":"ContainerStarted","Data":"3432ae0d5a4c7829facfd1b7032cc5b037ff1eb244ce4440937031c20387a824"} Nov 03 17:24:47.234043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.233608 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" event={"ID":"366af900-f825-402b-b013-b93794ec3c68","Type":"ContainerStarted","Data":"f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a"} Nov 03 17:24:47.303250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.303190 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" podStartSLOduration=2.303176734 podStartE2EDuration="2.303176734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:47.301893536 +0000 UTC m=+4182.635107109" watchObservedRunningTime="2026-11-03 17:24:47.303176734 +0000 UTC m=+4182.636390320" Nov 03 17:24:47.510177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.510145 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7"] Nov 03 17:24:47.510806 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:24:47.510776 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71f63d7_68a1_4cc8_a6ca_3a3bbb0964c6.slice/crio-7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665 WatchSource:0}: Error finding container 7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665: Status 404 returned error can't find the container with id 7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665 Nov 03 17:24:47.707948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.707911 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:47.708131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:47.707971 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:48.269976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.269933 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" event={"ID":"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6","Type":"ContainerStarted","Data":"7a4f8315a97d996c7f13b8d3c17ae0b66a2ab28234d2df6f91e738a6854e8895"} Nov 03 17:24:48.270588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.270556 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" event={"ID":"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6","Type":"ContainerStarted","Data":"7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665"} Nov 03 17:24:48.273893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.273866 6001 generic.go:356] "Generic (PLEG): container finished" podID="366af900-f825-402b-b013-b93794ec3c68" containerID="3432ae0d5a4c7829facfd1b7032cc5b037ff1eb244ce4440937031c20387a824" exitCode=0 Nov 03 17:24:48.274030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.273946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" event={"ID":"366af900-f825-402b-b013-b93794ec3c68","Type":"ContainerDied","Data":"3432ae0d5a4c7829facfd1b7032cc5b037ff1eb244ce4440937031c20387a824"} Nov 03 17:24:48.314994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.314936 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" podStartSLOduration=2.31492185 podStartE2EDuration="2.31492185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:24:48.313090127 +0000 UTC m=+4183.646303696" watchObservedRunningTime="2026-11-03 17:24:48.31492185 +0000 UTC m=+4183.648135419" Nov 03 17:24:48.707402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.707364 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:48.707644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.707421 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:48.824544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:48.824499 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" containerID="cri-o://fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463" gracePeriod=25 Nov 03 17:24:49.323470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.318790 6001 generic.go:356] "Generic (PLEG): container finished" podID="a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" containerID="7a4f8315a97d996c7f13b8d3c17ae0b66a2ab28234d2df6f91e738a6854e8895" exitCode=0 Nov 03 17:24:49.323470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.319821 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" event={"ID":"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6","Type":"ContainerDied","Data":"7a4f8315a97d996c7f13b8d3c17ae0b66a2ab28234d2df6f91e738a6854e8895"} Nov 03 17:24:49.620618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.620587 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:49.707526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.707480 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:49.707719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.707546 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:49.711102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.711050 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\") pod \"366af900-f825-402b-b013-b93794ec3c68\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " Nov 03 17:24:49.711285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.711166 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\" (UniqueName: \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\") pod \"366af900-f825-402b-b013-b93794ec3c68\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " Nov 03 17:24:49.711285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.711209 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\") pod \"366af900-f825-402b-b013-b93794ec3c68\" (UID: \"366af900-f825-402b-b013-b93794ec3c68\") " Nov 03 17:24:49.711680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.711652 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca" pod "366af900-f825-402b-b013-b93794ec3c68" (UID: "366af900-f825-402b-b013-b93794ec3c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:49.712236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.712210 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366af900-f825-402b-b013-b93794ec3c68-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:49.713536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.713506 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca" pod "366af900-f825-402b-b013-b93794ec3c68" (UID: "366af900-f825-402b-b013-b93794ec3c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:49.713809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.713784 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d" pod "366af900-f825-402b-b013-b93794ec3c68" (UID: "366af900-f825-402b-b013-b93794ec3c68"). InnerVolumeSpecName "kube-api-access-pph9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:49.814252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.814208 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366af900-f825-402b-b013-b93794ec3c68-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:49.814252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:49.814253 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pph9d\" (UniqueName: \"kubernetes.io/projected/366af900-f825-402b-b013-b93794ec3c68-kube-api-access-pph9d\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.333908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.333872 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" event={"ID":"366af900-f825-402b-b013-b93794ec3c68","Type":"ContainerDied","Data":"f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a"} Nov 03 17:24:50.333908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.333906 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m" Nov 03 17:24:50.334562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.333927 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6490d7e0e50de5f6b6830b62d487f48bbbc2d85b78639cfa99df84d4e4f012a" Nov 03 17:24:50.594607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.594525 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:50.739468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.731046 6001 patch_prober.go:29] interesting pod/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" start-of-body= Nov 03 17:24:50.739468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.731116 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.117:15021/healthz/ready\": dial tcp 10.134.1.117:15021: connect: connection refused" Nov 03 17:24:50.773471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.765177 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\") pod \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " Nov 03 17:24:50.773471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.765390 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\") pod \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " Nov 03 17:24:50.773471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.765631 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\") pod \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\" (UID: \"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6\") " Nov 03 17:24:50.773471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.769677 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca" pod "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" (UID: "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:50.791723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.791673 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l" pod "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" (UID: "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6"). InnerVolumeSpecName "kube-api-access-9hk6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:50.818648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.818596 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca" pod "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" (UID: "a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:50.868721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.868630 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.868721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.868662 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.868721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.868673 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hk6l\" (UniqueName: \"kubernetes.io/projected/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6-kube-api-access-9hk6l\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.874110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:50.874072 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:51.344208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.344166 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" event={"ID":"a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6","Type":"ContainerDied","Data":"7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665"} Nov 03 17:24:51.344753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.344235 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fa573c49341efefdca81bd58107e5d98cd37ad42364ced9a75b55082be5b665" Nov 03 17:24:51.344753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.344182 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7" Nov 03 17:24:51.713850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.713745 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:51.714297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.714271 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:51.715140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:51.715110 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:24:52.253567 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:24:52.253527 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:24:57.315499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.315456 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8"] Nov 03 17:24:57.316516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316494 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" containerName="revoke-keys" Nov 03 17:24:57.316516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316516 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" containerName="revoke-keys" Nov 03 17:24:57.316643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316572 6001 cpu_manager.go:414] "containerMap: removing container" podUID="366af900-f825-402b-b013-b93794ec3c68" containerName="revoke-keys" Nov 03 17:24:57.316643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316578 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366af900-f825-402b-b013-b93794ec3c68" containerName="revoke-keys" Nov 03 17:24:57.316740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316731 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="366af900-f825-402b-b013-b93794ec3c68" containerName="revoke-keys" Nov 03 17:24:57.316782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.316747 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" containerName="revoke-keys" Nov 03 17:24:57.321323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.321301 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.341062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.341032 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8"] Nov 03 17:24:57.348497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.348455 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:24:57.348695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.348503 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:24:57.348801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.348529 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:24:57.351080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.351057 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:24:57.429471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.428940 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.429471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.429063 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.429471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.429162 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjwf\" (UniqueName: \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.530700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.530652 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.530924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.530774 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjwf\" (UniqueName: \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.531107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.531089 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.531340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.531317 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.533718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.533695 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.557818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.557775 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjwf\" (UniqueName: \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\") pod \"maas-api-revoke-sub-0d4bb8b2-vb5k8\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.639476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.639385 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:24:57.955805 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:24:57.955770 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2af11a6_30f1_41b4_857f_25967bc2785d.slice/crio-c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7 WatchSource:0}: Error finding container c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7: Status 404 returned error can't find the container with id c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7 Nov 03 17:24:57.957052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:57.957020 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8"] Nov 03 17:24:58.486094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:58.486056 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" event={"ID":"c2af11a6-30f1-41b4-857f-25967bc2785d","Type":"ContainerStarted","Data":"c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7"} Nov 03 17:24:59.496739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:59.496699 6001 generic.go:356] "Generic (PLEG): container finished" podID="c2af11a6-30f1-41b4-857f-25967bc2785d" containerID="1dc3019421b168cc03a6b2fd0388e9ce659b5e67cfce306557093cfb6fb85c86" exitCode=0 Nov 03 17:24:59.497170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:24:59.496782 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" event={"ID":"c2af11a6-30f1-41b4-857f-25967bc2785d","Type":"ContainerDied","Data":"1dc3019421b168cc03a6b2fd0388e9ce659b5e67cfce306557093cfb6fb85c86"} Nov 03 17:25:00.737044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.737012 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:25:00.874303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.874207 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:00.916948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.916908 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\" (UniqueName: \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\") pod \"c2af11a6-30f1-41b4-857f-25967bc2785d\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " Nov 03 17:25:00.917158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.917032 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\") pod \"c2af11a6-30f1-41b4-857f-25967bc2785d\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " Nov 03 17:25:00.917249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.917193 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\") pod \"c2af11a6-30f1-41b4-857f-25967bc2785d\" (UID: \"c2af11a6-30f1-41b4-857f-25967bc2785d\") " Nov 03 17:25:00.917484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.917429 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca" pod "c2af11a6-30f1-41b4-857f-25967bc2785d" (UID: "c2af11a6-30f1-41b4-857f-25967bc2785d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:00.917842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.917825 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:00.919258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.919226 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf" pod "c2af11a6-30f1-41b4-857f-25967bc2785d" (UID: "c2af11a6-30f1-41b4-857f-25967bc2785d"). InnerVolumeSpecName "kube-api-access-7zjwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:00.919258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:00.919244 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca" pod "c2af11a6-30f1-41b4-857f-25967bc2785d" (UID: "c2af11a6-30f1-41b4-857f-25967bc2785d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:01.019241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:01.019197 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2af11a6-30f1-41b4-857f-25967bc2785d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:01.019241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:01.019235 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjwf\" (UniqueName: \"kubernetes.io/projected/c2af11a6-30f1-41b4-857f-25967bc2785d-kube-api-access-7zjwf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:01.519400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:01.519369 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" Nov 03 17:25:01.519586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:01.519404 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8" event={"ID":"c2af11a6-30f1-41b4-857f-25967bc2785d","Type":"ContainerDied","Data":"c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7"} Nov 03 17:25:01.519586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:01.519472 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b4fece451a61ed8374e4d7cd66f960ae83bd3e8ffbf5a7bb0f55e6a2d869c7" Nov 03 17:25:05.691268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:05.691235 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:25:05.803269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:05.803242 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:25:09.134569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.134540 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:25:09.186371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.186333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\") pod \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " Nov 03 17:25:09.186593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.186458 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\" (UniqueName: \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\") pod \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\" (UID: \"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f\") " Nov 03 17:25:09.186790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.186767 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume" pod "d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" (UID: "d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:09.187218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.187185 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:09.188765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.188731 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6" pod "d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" (UID: "d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f"). InnerVolumeSpecName "kube-api-access-qzhc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:09.293363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.293320 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzhc6\" (UniqueName: \"kubernetes.io/projected/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f-kube-api-access-qzhc6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:09.599101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.599067 6001 generic.go:356] "Generic (PLEG): container finished" podID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerID="fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463" exitCode=0 Nov 03 17:25:09.599311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.599116 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" Nov 03 17:25:09.599311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.599160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" event={"ID":"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f","Type":"ContainerDied","Data":"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463"} Nov 03 17:25:09.599311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.599212 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5" event={"ID":"d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f","Type":"ContainerDied","Data":"f7d8f4bfbe8cc2918c3d33071a53042140d1f0f0ef07a0ceb343dedec6a3f81f"} Nov 03 17:25:09.599311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.599239 6001 scope.go:122] "RemoveContainer" containerID="fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463" Nov 03 17:25:09.612161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.612135 6001 scope.go:122] "RemoveContainer" containerID="fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463" Nov 03 17:25:09.612530 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:09.612507 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463\": container with ID starting with fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463 not found: ID does not exist" containerID="fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463" Nov 03 17:25:09.612623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.612540 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463"} err="failed to get container status \"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463\": rpc error: code = NotFound desc = could not find container \"fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463\": container with ID starting with fd6edf64a2e8a3d89a122dc9b102294a7534a59ce70e0352b5817b021cd5e463 not found: ID does not exist" Nov 03 17:25:09.657000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.656965 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:25:09.668962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:09.668932 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-9c4762-6f5784cdb9-55dv5"] Nov 03 17:25:10.471186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:10.471151 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:25:10.471716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:10.471432 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" containerID="cri-o://b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05" gracePeriod=30 Nov 03 17:25:10.817507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:10.816831 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" path="/var/lib/kubelet/pods/d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f/volumes" Nov 03 17:25:13.141336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:13.141251 6001 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:25:13.141336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:13.141322 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:25:14.460072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.460035 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6"] Nov 03 17:25:14.461019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461002 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c2af11a6-30f1-41b4-857f-25967bc2785d" containerName="revoke-keys" Nov 03 17:25:14.461091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461021 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2af11a6-30f1-41b4-857f-25967bc2785d" containerName="revoke-keys" Nov 03 17:25:14.461091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461047 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" Nov 03 17:25:14.461091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461053 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" Nov 03 17:25:14.461205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461196 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2af11a6-30f1-41b4-857f-25967bc2785d" containerName="revoke-keys" Nov 03 17:25:14.461248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.461210 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a0f4c2-d6b3-41d2-bec3-29829c5bfe8f" containerName="payload-processing" Nov 03 17:25:14.465389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.465373 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.484093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.484064 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6"] Nov 03 17:25:14.491136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.491105 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:25:14.491360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.491338 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:25:14.496678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.496647 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:25:14.496831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.496814 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:25:14.627844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.627806 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.628039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.627947 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.628140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.628122 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.730487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.730405 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.731299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.731275 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.731718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.731638 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.733418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.733386 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.736353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.736324 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.755004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.754977 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\") pod \"maas-api-revoke-sub-b586ec8c-t8gl6\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:14.782667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:14.782263 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:15.102799 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:25:15.102768 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88369823_5e2d_4a70_ae36_83b1d47d5114.slice/crio-dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b WatchSource:0}: Error finding container dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b: Status 404 returned error can't find the container with id dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b Nov 03 17:25:15.103541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.103508 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6"] Nov 03 17:25:15.740365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.740327 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:25:15.806280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.806243 6001 generic.go:356] "Generic (PLEG): container finished" podID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerID="b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05" exitCode=0 Nov 03 17:25:15.806500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.806293 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" Nov 03 17:25:15.806500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.806327 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" event={"ID":"6a00a1fe-17ef-4200-9d20-c999c2f92516","Type":"ContainerDied","Data":"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05"} Nov 03 17:25:15.806500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.806390 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq" event={"ID":"6a00a1fe-17ef-4200-9d20-c999c2f92516","Type":"ContainerDied","Data":"1b091ccf068ee5371ca793c05e2bd4fc683cb8783761d0d5b4eb6dfcbf3c990d"} Nov 03 17:25:15.806500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.806417 6001 scope.go:122] "RemoveContainer" containerID="b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05" Nov 03 17:25:15.808317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.808290 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" event={"ID":"88369823-5e2d-4a70-ae36-83b1d47d5114","Type":"ContainerStarted","Data":"61d2b7c6029fa59652314ea23927157b01f7a75f88a22d4e6542ccf9973e8c05"} Nov 03 17:25:15.808426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.808341 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" event={"ID":"88369823-5e2d-4a70-ae36-83b1d47d5114","Type":"ContainerStarted","Data":"dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b"} Nov 03 17:25:15.820151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.820131 6001 scope.go:122] "RemoveContainer" containerID="b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05" Nov 03 17:25:15.820542 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:15.820462 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05\": container with ID starting with b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05 not found: ID does not exist" containerID="b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05" Nov 03 17:25:15.820542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.820499 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05"} err="failed to get container status \"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05\": rpc error: code = NotFound desc = could not find container \"b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05\": container with ID starting with b52813ddf6fef738e2cad9803b8e04eecd6347e1dfd67a4b9bce924f552d5e05 not found: ID does not exist" Nov 03 17:25:15.850724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.850690 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.850929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.850774 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.850929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.850897 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851071 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851085 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.851246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851117 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851163 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851225 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851270 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851407 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\") pod \"6a00a1fe-17ef-4200-9d20-c999c2f92516\" (UID: \"6a00a1fe-17ef-4200-9d20-c999c2f92516\") " Nov 03 17:25:15.851707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851662 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.852175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.851885 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.852351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.852311 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.852351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.852333 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.852351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.852347 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.852638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.852366 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:15.852638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.852427 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.854464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.854424 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:25:15.854765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.854740 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "kube-api-access-pbzk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:15.855018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.854994 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:15.856570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.856542 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy" pod "6a00a1fe-17ef-4200-9d20-c999c2f92516" (UID: "6a00a1fe-17ef-4200-9d20-c999c2f92516"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.860572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.860526 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" podStartSLOduration=1.86051452 podStartE2EDuration="1.86051452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:25:15.859277706 +0000 UTC m=+4211.192491275" watchObservedRunningTime="2026-11-03 17:25:15.86051452 +0000 UTC m=+4211.193728088" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953780 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a00a1fe-17ef-4200-9d20-c999c2f92516-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953806 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbzk6\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-kube-api-access-pbzk6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953818 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953832 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953841 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.953851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:15.953849 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a00a1fe-17ef-4200-9d20-c999c2f92516-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:16.158625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.158578 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:25:16.176495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.169673 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-drjmq"] Nov 03 17:25:16.558931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.558892 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:25:16.600095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.600055 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:25:16.792740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.792706 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" path="/var/lib/kubelet/pods/6a00a1fe-17ef-4200-9d20-c999c2f92516/volumes" Nov 03 17:25:16.864838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.864747 6001 generic.go:356] "Generic (PLEG): container finished" podID="88369823-5e2d-4a70-ae36-83b1d47d5114" containerID="61d2b7c6029fa59652314ea23927157b01f7a75f88a22d4e6542ccf9973e8c05" exitCode=0 Nov 03 17:25:16.865012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:16.864833 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" event={"ID":"88369823-5e2d-4a70-ae36-83b1d47d5114","Type":"ContainerDied","Data":"61d2b7c6029fa59652314ea23927157b01f7a75f88a22d4e6542ccf9973e8c05"} Nov 03 17:25:18.074993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.074962 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:18.105324 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.105294 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\") pod \"88369823-5e2d-4a70-ae36-83b1d47d5114\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " Nov 03 17:25:18.105606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.105339 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\") pod \"88369823-5e2d-4a70-ae36-83b1d47d5114\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " Nov 03 17:25:18.105606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.105430 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\") pod \"88369823-5e2d-4a70-ae36-83b1d47d5114\" (UID: \"88369823-5e2d-4a70-ae36-83b1d47d5114\") " Nov 03 17:25:18.105768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.105725 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca" pod "88369823-5e2d-4a70-ae36-83b1d47d5114" (UID: "88369823-5e2d-4a70-ae36-83b1d47d5114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:18.106182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.106164 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:18.107422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.107399 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca" pod "88369823-5e2d-4a70-ae36-83b1d47d5114" (UID: "88369823-5e2d-4a70-ae36-83b1d47d5114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:18.107599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.107512 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8" pod "88369823-5e2d-4a70-ae36-83b1d47d5114" (UID: "88369823-5e2d-4a70-ae36-83b1d47d5114"). InnerVolumeSpecName "kube-api-access-8bfb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:18.208373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.208287 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bfb8\" (UniqueName: \"kubernetes.io/projected/88369823-5e2d-4a70-ae36-83b1d47d5114-kube-api-access-8bfb8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:18.208373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.208320 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88369823-5e2d-4a70-ae36-83b1d47d5114-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:18.884118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.884089 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" Nov 03 17:25:18.884282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.884176 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6" event={"ID":"88369823-5e2d-4a70-ae36-83b1d47d5114","Type":"ContainerDied","Data":"dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b"} Nov 03 17:25:18.884282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:18.884220 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dce96158e41f0c251285e22567216e618b17a124c1d8804655b39746662e604b" Nov 03 17:25:21.559701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:21.559651 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" containerID="cri-o://1c4fc2e46920969418d1f3c8f29e98697a09ef9efddb854b8c996790fbb43737" gracePeriod=25 Nov 03 17:25:21.628473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:21.627524 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" containerID="cri-o://09a407cfff16dbccd4734b028cf7cc815829a18b02acb3e5226d6c4e04c5b9a4" gracePeriod=25 Nov 03 17:25:22.253201 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:22.253158 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:25:26.657544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:26.657499 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:25:26.684621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:26.684586 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:25:29.529898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:29.529858 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:30.540747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:30.540705 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:31.658671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:31.658629 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" podUID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" containerName="payload-processing" containerID="cri-o://685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5" gracePeriod=25 Nov 03 17:25:31.685754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:31.685709 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" containerID="cri-o://c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f" gracePeriod=25 Nov 03 17:25:31.984332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:31.984304 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:25:32.065651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.065612 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\") pod \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " Nov 03 17:25:32.065865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.065712 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\" (UniqueName: \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\") pod \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\" (UID: \"fab40dc7-5ae9-4a08-9c01-f15227c21f82\") " Nov 03 17:25:32.066136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.066107 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume" pod "fab40dc7-5ae9-4a08-9c01-f15227c21f82" (UID: "fab40dc7-5ae9-4a08-9c01-f15227c21f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:32.066653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.066626 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab40dc7-5ae9-4a08-9c01-f15227c21f82-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:32.068800 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.068771 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6" pod "fab40dc7-5ae9-4a08-9c01-f15227c21f82" (UID: "fab40dc7-5ae9-4a08-9c01-f15227c21f82"). InnerVolumeSpecName "kube-api-access-cd9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:32.083248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.083214 6001 generic.go:356] "Generic (PLEG): container finished" podID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" containerID="685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5" exitCode=0 Nov 03 17:25:32.083501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.083277 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" Nov 03 17:25:32.083501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.083304 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" event={"ID":"fab40dc7-5ae9-4a08-9c01-f15227c21f82","Type":"ContainerDied","Data":"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5"} Nov 03 17:25:32.083501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.083369 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn" event={"ID":"fab40dc7-5ae9-4a08-9c01-f15227c21f82","Type":"ContainerDied","Data":"6b4a17439b1724c8f72fa44ec6728ae5e3651c6f41cc7262f5ea215948e864e1"} Nov 03 17:25:32.083501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.083401 6001 scope.go:122] "RemoveContainer" containerID="685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5" Nov 03 17:25:32.100257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.100238 6001 scope.go:122] "RemoveContainer" containerID="685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5" Nov 03 17:25:32.100642 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:32.100621 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5\": container with ID starting with 685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5 not found: ID does not exist" containerID="685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5" Nov 03 17:25:32.100730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.100651 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5"} err="failed to get container status \"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5\": rpc error: code = NotFound desc = could not find container \"685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5\": container with ID starting with 685eafa5456d4e46327678b3262522181d130811d8fefea5434e75b1b0c1cbb5 not found: ID does not exist" Nov 03 17:25:32.141057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.141021 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:25:32.145437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.145401 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-44736d-68b4677c6f-gfwqn"] Nov 03 17:25:32.168176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.168136 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd9c6\" (UniqueName: \"kubernetes.io/projected/fab40dc7-5ae9-4a08-9c01-f15227c21f82-kube-api-access-cd9c6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:32.792290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:32.792254 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" path="/var/lib/kubelet/pods/fab40dc7-5ae9-4a08-9c01-f15227c21f82/volumes" Nov 03 17:25:35.889655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:35.889586 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:25:35.890307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:35.890253 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" containerID="cri-o://bb3f77f1684cb201ac8829820cf1aaac6fa2446a86e69fb3e20786377b8d6ab3" gracePeriod=30 Nov 03 17:25:37.558617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:37.558564 6001 patch_prober.go:29] interesting pod/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:25:37.559050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:37.558647 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:25:38.394668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:38.394632 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:25:38.429344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:38.429306 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:25:39.246200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:39.246154 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:39.529543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:39.529427 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:40.540872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:40.540830 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:41.337086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.335965 6001 generic.go:356] "Generic (PLEG): container finished" podID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerID="bb3f77f1684cb201ac8829820cf1aaac6fa2446a86e69fb3e20786377b8d6ab3" exitCode=0 Nov 03 17:25:41.337086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.336124 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerDied","Data":"bb3f77f1684cb201ac8829820cf1aaac6fa2446a86e69fb3e20786377b8d6ab3"} Nov 03 17:25:41.337086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.336179 6001 scope.go:122] "RemoveContainer" containerID="9eaf88450cc28e923fc33028de10e815162fc3b49c8756cf93e8af1f223bf9fc" Nov 03 17:25:41.426638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.426605 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:25:41.502369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502316 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502499 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502537 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502611 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502656 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502730 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.502865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502832 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.503034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.502888 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.503034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.503007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\") pod \"3529aefc-dbb5-415b-9192-b233f5cf4cba\" (UID: \"3529aefc-dbb5-415b-9192-b233f5cf4cba\") " Nov 03 17:25:41.504056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.504022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:41.504807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.504727 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:41.504807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.504757 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:41.505251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.505161 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:41.505480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.505417 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:41.507153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.507126 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:41.514548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.508619 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:41.514548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.508752 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:25:41.514548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.512990 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl" pod "3529aefc-dbb5-415b-9192-b233f5cf4cba" (UID: "3529aefc-dbb5-415b-9192-b233f5cf4cba"). InnerVolumeSpecName "kube-api-access-ccrkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:41.606025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.605980 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606022 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606036 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606049 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrkl\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-kube-api-access-ccrkl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606062 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3529aefc-dbb5-415b-9192-b233f5cf4cba-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606075 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606086 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606098 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:41.606565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:41.606108 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3529aefc-dbb5-415b-9192-b233f5cf4cba-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:42.384997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.372268 6001 generic.go:356] "Generic (PLEG): container finished" podID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerID="1c4fc2e46920969418d1f3c8f29e98697a09ef9efddb854b8c996790fbb43737" exitCode=0 Nov 03 17:25:42.384997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.372425 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" event={"ID":"93ac7f38-1526-4247-8d50-9625702dd2dd","Type":"ContainerDied","Data":"1c4fc2e46920969418d1f3c8f29e98697a09ef9efddb854b8c996790fbb43737"} Nov 03 17:25:42.384997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.374422 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" event={"ID":"3529aefc-dbb5-415b-9192-b233f5cf4cba","Type":"ContainerDied","Data":"71a01ddd0cec4ff30d25631feddbb1f8a823fcb5d89f51a5a39870704d1dadd0"} Nov 03 17:25:42.384997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.374502 6001 scope.go:122] "RemoveContainer" containerID="bb3f77f1684cb201ac8829820cf1aaac6fa2446a86e69fb3e20786377b8d6ab3" Nov 03 17:25:42.384997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.374698 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm" Nov 03 17:25:42.430180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.428480 6001 generic.go:356] "Generic (PLEG): container finished" podID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerID="09a407cfff16dbccd4734b028cf7cc815829a18b02acb3e5226d6c4e04c5b9a4" exitCode=0 Nov 03 17:25:42.430180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.428558 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" event={"ID":"2e754262-de1f-45d8-8d8a-dab2b7985a22","Type":"ContainerDied","Data":"09a407cfff16dbccd4734b028cf7cc815829a18b02acb3e5226d6c4e04c5b9a4"} Nov 03 17:25:42.518408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.518342 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:25:42.526533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.526507 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:25:42.545343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.545304 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:25:42.559393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.559359 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9c4762-openshift-default-7cbd574df5-fmtmm"] Nov 03 17:25:42.624433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.623588 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\" (UniqueName: \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\") pod \"2e754262-de1f-45d8-8d8a-dab2b7985a22\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " Nov 03 17:25:42.624433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.624205 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\") pod \"2e754262-de1f-45d8-8d8a-dab2b7985a22\" (UID: \"2e754262-de1f-45d8-8d8a-dab2b7985a22\") " Nov 03 17:25:42.625141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.624558 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\" (UniqueName: \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\") pod \"93ac7f38-1526-4247-8d50-9625702dd2dd\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " Nov 03 17:25:42.625141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.624653 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\") pod \"93ac7f38-1526-4247-8d50-9625702dd2dd\" (UID: \"93ac7f38-1526-4247-8d50-9625702dd2dd\") " Nov 03 17:25:42.625141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.624673 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume" pod "2e754262-de1f-45d8-8d8a-dab2b7985a22" (UID: "2e754262-de1f-45d8-8d8a-dab2b7985a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:42.626211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.625838 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e754262-de1f-45d8-8d8a-dab2b7985a22-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:42.626687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.626658 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume" pod "93ac7f38-1526-4247-8d50-9625702dd2dd" (UID: "93ac7f38-1526-4247-8d50-9625702dd2dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:42.767467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.756137 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ac7f38-1526-4247-8d50-9625702dd2dd-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:42.846467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.841612 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9" pod "2e754262-de1f-45d8-8d8a-dab2b7985a22" (UID: "2e754262-de1f-45d8-8d8a-dab2b7985a22"). InnerVolumeSpecName "kube-api-access-s5rf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:42.846467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.843665 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb" pod "93ac7f38-1526-4247-8d50-9625702dd2dd" (UID: "93ac7f38-1526-4247-8d50-9625702dd2dd"). InnerVolumeSpecName "kube-api-access-ltrgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:42.864465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.859437 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5rf9\" (UniqueName: \"kubernetes.io/projected/2e754262-de1f-45d8-8d8a-dab2b7985a22-kube-api-access-s5rf9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:42.864465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.859513 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrgb\" (UniqueName: \"kubernetes.io/projected/93ac7f38-1526-4247-8d50-9625702dd2dd-kube-api-access-ltrgb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:42.881376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:42.881337 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" path="/var/lib/kubelet/pods/3529aefc-dbb5-415b-9192-b233f5cf4cba/volumes" Nov 03 17:25:43.395182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.395097 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" containerID="cri-o://0968bc25fdae5074dad7ba8f5456b6387c23d39bc9b072126d8136ed6ac4d24d" gracePeriod=25 Nov 03 17:25:43.430328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.430277 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" containerID="cri-o://61d9bf927879183c2435ceda76a16629c04beea925a964a38631d371809b973a" gracePeriod=25 Nov 03 17:25:43.441831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.441799 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" Nov 03 17:25:43.441831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.441816 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6" event={"ID":"2e754262-de1f-45d8-8d8a-dab2b7985a22","Type":"ContainerDied","Data":"ee0c852f49ecba5d0b17f9b115c925446ddf7a1a99b04f71422b73c5682b2d41"} Nov 03 17:25:43.442082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.441887 6001 scope.go:122] "RemoveContainer" containerID="09a407cfff16dbccd4734b028cf7cc815829a18b02acb3e5226d6c4e04c5b9a4" Nov 03 17:25:43.443985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.443952 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" Nov 03 17:25:43.444125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.443995 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn" event={"ID":"93ac7f38-1526-4247-8d50-9625702dd2dd","Type":"ContainerDied","Data":"34f380139b01d3eb3a574d62c364830e209bc11c49d6d7ed12451f2239a656f1"} Nov 03 17:25:43.455814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.455789 6001 scope.go:122] "RemoveContainer" containerID="1c4fc2e46920969418d1f3c8f29e98697a09ef9efddb854b8c996790fbb43737" Nov 03 17:25:43.528859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.528814 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:25:43.566358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.566323 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-527051-76d95855c9-ljtj6"] Nov 03 17:25:43.629133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.629096 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:25:43.633529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:43.633503 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-527051-788c8b4495-zccjn"] Nov 03 17:25:44.801262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:44.801224 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" path="/var/lib/kubelet/pods/2e754262-de1f-45d8-8d8a-dab2b7985a22/volumes" Nov 03 17:25:44.802869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:44.802845 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" path="/var/lib/kubelet/pods/93ac7f38-1526-4247-8d50-9625702dd2dd/volumes" Nov 03 17:25:49.206561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:49.203867 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:51.727701 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:51.727666 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f71b6a_a26b_4648_8388_7e28f656537a.slice/crio-c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:25:51.997914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:51.997885 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:25:52.062950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.062920 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\" (UniqueName: \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\") pod \"22f71b6a-a26b-4648-8388-7e28f656537a\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " Nov 03 17:25:52.063138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.063123 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\") pod \"22f71b6a-a26b-4648-8388-7e28f656537a\" (UID: \"22f71b6a-a26b-4648-8388-7e28f656537a\") " Nov 03 17:25:52.063600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.063568 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume" pod "22f71b6a-a26b-4648-8388-7e28f656537a" (UID: "22f71b6a-a26b-4648-8388-7e28f656537a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:52.063770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.063752 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22f71b6a-a26b-4648-8388-7e28f656537a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:52.073751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.073722 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j" pod "22f71b6a-a26b-4648-8388-7e28f656537a" (UID: "22f71b6a-a26b-4648-8388-7e28f656537a"). InnerVolumeSpecName "kube-api-access-56v6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:52.165250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.165213 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56v6j\" (UniqueName: \"kubernetes.io/projected/22f71b6a-a26b-4648-8388-7e28f656537a-kube-api-access-56v6j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:25:52.252590 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:52.252554 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:25:52.342779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.342742 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:25:52.536297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.536203 6001 generic.go:356] "Generic (PLEG): container finished" podID="22f71b6a-a26b-4648-8388-7e28f656537a" containerID="c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f" exitCode=0 Nov 03 17:25:52.536297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.536248 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" event={"ID":"22f71b6a-a26b-4648-8388-7e28f656537a","Type":"ContainerDied","Data":"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f"} Nov 03 17:25:52.536297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.536287 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" event={"ID":"22f71b6a-a26b-4648-8388-7e28f656537a","Type":"ContainerDied","Data":"1c5f223f9a011570ab8d017b23ca68bef214f5ecbd1afc576ebc7cb8153221bd"} Nov 03 17:25:52.536642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.536309 6001 scope.go:122] "RemoveContainer" containerID="c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f" Nov 03 17:25:52.536642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.536251 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5" Nov 03 17:25:52.548740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.548717 6001 scope.go:122] "RemoveContainer" containerID="c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f" Nov 03 17:25:52.549058 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:25:52.549034 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f\": container with ID starting with c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f not found: ID does not exist" containerID="c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f" Nov 03 17:25:52.549138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.549064 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f"} err="failed to get container status \"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f\": rpc error: code = NotFound desc = could not find container \"c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f\": container with ID starting with c6765b452352b57b9ddf1d9fb5d843e4c4ff9031018992c83864b56805718a0f not found: ID does not exist" Nov 03 17:25:52.636664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.636625 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:25:52.642113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.642078 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-44736d-65df5f569f-lm8g5"] Nov 03 17:25:52.792691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:52.792604 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" path="/var/lib/kubelet/pods/22f71b6a-a26b-4648-8388-7e28f656537a/volumes" Nov 03 17:25:53.121527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:53.117338 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:53.121527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:53.117762 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:57.343056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:57.343005 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" containerID="cri-o://0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf" gracePeriod=25 Nov 03 17:25:57.469852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:25:57.469810 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:03.084083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.084045 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:03.084542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.084044 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:03.668742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.668642 6001 generic.go:356] "Generic (PLEG): container finished" podID="981fa68e-4312-47f6-8f40-8e96d706b102" containerID="0968bc25fdae5074dad7ba8f5456b6387c23d39bc9b072126d8136ed6ac4d24d" exitCode=0 Nov 03 17:26:03.668916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.668783 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" event={"ID":"981fa68e-4312-47f6-8f40-8e96d706b102","Type":"ContainerDied","Data":"0968bc25fdae5074dad7ba8f5456b6387c23d39bc9b072126d8136ed6ac4d24d"} Nov 03 17:26:03.672633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.672316 6001 generic.go:356] "Generic (PLEG): container finished" podID="904551aa-b8b4-479e-b24b-247fec59c56d" containerID="61d9bf927879183c2435ceda76a16629c04beea925a964a38631d371809b973a" exitCode=0 Nov 03 17:26:03.672633 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.672564 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" event={"ID":"904551aa-b8b4-479e-b24b-247fec59c56d","Type":"ContainerDied","Data":"61d9bf927879183c2435ceda76a16629c04beea925a964a38631d371809b973a"} Nov 03 17:26:03.750864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.750834 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:26:03.786252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.786227 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:26:03.904831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.904791 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\" (UniqueName: \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\") pod \"981fa68e-4312-47f6-8f40-8e96d706b102\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " Nov 03 17:26:03.905021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.904940 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\" (UniqueName: \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\") pod \"904551aa-b8b4-479e-b24b-247fec59c56d\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " Nov 03 17:26:03.905135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905116 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\") pod \"981fa68e-4312-47f6-8f40-8e96d706b102\" (UID: \"981fa68e-4312-47f6-8f40-8e96d706b102\") " Nov 03 17:26:03.905185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905155 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\") pod \"904551aa-b8b4-479e-b24b-247fec59c56d\" (UID: \"904551aa-b8b4-479e-b24b-247fec59c56d\") " Nov 03 17:26:03.905555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905523 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume" pod "981fa68e-4312-47f6-8f40-8e96d706b102" (UID: "981fa68e-4312-47f6-8f40-8e96d706b102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:03.905707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905606 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume" pod "904551aa-b8b4-479e-b24b-247fec59c56d" (UID: "904551aa-b8b4-479e-b24b-247fec59c56d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:03.905909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905887 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/981fa68e-4312-47f6-8f40-8e96d706b102-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:03.905962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.905915 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/904551aa-b8b4-479e-b24b-247fec59c56d-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:03.907051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.907029 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb" pod "981fa68e-4312-47f6-8f40-8e96d706b102" (UID: "981fa68e-4312-47f6-8f40-8e96d706b102"). InnerVolumeSpecName "kube-api-access-98cnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:03.907165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:03.907147 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94" pod "904551aa-b8b4-479e-b24b-247fec59c56d" (UID: "904551aa-b8b4-479e-b24b-247fec59c56d"). InnerVolumeSpecName "kube-api-access-cvp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:04.007056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.007014 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98cnb\" (UniqueName: \"kubernetes.io/projected/981fa68e-4312-47f6-8f40-8e96d706b102-kube-api-access-98cnb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:04.007056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.007050 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvp94\" (UniqueName: \"kubernetes.io/projected/904551aa-b8b4-479e-b24b-247fec59c56d-kube-api-access-cvp94\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:04.682917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.682826 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" Nov 03 17:26:04.683378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.682828 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx" event={"ID":"981fa68e-4312-47f6-8f40-8e96d706b102","Type":"ContainerDied","Data":"9da725249a3aa89f599d0ca15e990ca74d8234f52c4773ffa001acdbae14237e"} Nov 03 17:26:04.683378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.683004 6001 scope.go:122] "RemoveContainer" containerID="0968bc25fdae5074dad7ba8f5456b6387c23d39bc9b072126d8136ed6ac4d24d" Nov 03 17:26:04.685054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.685033 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" Nov 03 17:26:04.685188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.685164 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt" event={"ID":"904551aa-b8b4-479e-b24b-247fec59c56d","Type":"ContainerDied","Data":"2808121bb56fd23afa38b8e78994754c42d512c816fc186235aacf2d38e32a41"} Nov 03 17:26:04.696866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.696840 6001 scope.go:122] "RemoveContainer" containerID="61d9bf927879183c2435ceda76a16629c04beea925a964a38631d371809b973a" Nov 03 17:26:04.740007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.739974 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:26:04.752419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.752374 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cb0fa3-776d997f5-ghztx"] Nov 03 17:26:04.796214 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.796180 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" path="/var/lib/kubelet/pods/981fa68e-4312-47f6-8f40-8e96d706b102/volumes" Nov 03 17:26:04.796679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.796661 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:26:04.796739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:04.796694 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cb0fa3-7ffd8578f5-xvqwt"] Nov 03 17:26:05.956031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.955983 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84"] Nov 03 17:26:05.956915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956896 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956916 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956939 6001 cpu_manager.go:414] "containerMap: removing container" podUID="88369823-5e2d-4a70-ae36-83b1d47d5114" containerName="revoke-keys" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956945 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88369823-5e2d-4a70-ae36-83b1d47d5114" containerName="revoke-keys" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956956 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956962 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956983 6001 cpu_manager.go:414] "containerMap: removing container" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956988 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.956996 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" Nov 03 17:26:05.957000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957001 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957013 6001 cpu_manager.go:414] "containerMap: removing container" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957019 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957028 6001 cpu_manager.go:414] "containerMap: removing container" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957033 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957044 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957049 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957057 6001 cpu_manager.go:414] "containerMap: removing container" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957061 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957192 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957200 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="88369823-5e2d-4a70-ae36-83b1d47d5114" containerName="revoke-keys" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957208 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e754262-de1f-45d8-8d8a-dab2b7985a22" containerName="payload-pre-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957219 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" containerName="payload-pre-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957227 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a00a1fe-17ef-4200-9d20-c999c2f92516" containerName="istio-proxy" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957235 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="981fa68e-4312-47f6-8f40-8e96d706b102" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957241 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f71b6a-a26b-4648-8388-7e28f656537a" containerName="payload-pre-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957248 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab40dc7-5ae9-4a08-9c01-f15227c21f82" containerName="payload-processing" Nov 03 17:26:05.957506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.957254 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ac7f38-1526-4247-8d50-9625702dd2dd" containerName="payload-processing" Nov 03 17:26:05.962733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.962711 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:05.979156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.979125 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:26:05.983717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.983693 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84"] Nov 03 17:26:05.991615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.991553 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:26:05.991615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.991562 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:26:05.991615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:05.991585 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:26:06.042276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.042235 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.042512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.042361 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp624\" (UniqueName: \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.042512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.042508 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.144459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.144405 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.144664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.144623 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.144744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.144726 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp624\" (UniqueName: \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.145158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.145135 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.147100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.147083 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.169812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.169781 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp624\" (UniqueName: \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\") pod \"maas-api-revoke-sub-0b8cbf0f-mjc84\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.278750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.278700 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:06.589470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.589425 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84"] Nov 03 17:26:06.590551 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:26:06.590526 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1190400_8d4a_4943_acd3_6d56fd59d03d.slice/crio-0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe WatchSource:0}: Error finding container 0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe: Status 404 returned error can't find the container with id 0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe Nov 03 17:26:06.707291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.707253 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" event={"ID":"d1190400-8d4a-4943-acd3-6d56fd59d03d","Type":"ContainerStarted","Data":"0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe"} Nov 03 17:26:06.791756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:06.791719 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904551aa-b8b4-479e-b24b-247fec59c56d" path="/var/lib/kubelet/pods/904551aa-b8b4-479e-b24b-247fec59c56d/volumes" Nov 03 17:26:07.469778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:07.469736 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:07.719534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:07.719480 6001 generic.go:356] "Generic (PLEG): container finished" podID="d1190400-8d4a-4943-acd3-6d56fd59d03d" containerID="7d48b44b3630f2d98d087609dd5b8cd10144975f9c5ad6492b085b775cfa1cfd" exitCode=0 Nov 03 17:26:07.719735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:07.719566 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" event={"ID":"d1190400-8d4a-4943-acd3-6d56fd59d03d","Type":"ContainerDied","Data":"7d48b44b3630f2d98d087609dd5b8cd10144975f9c5ad6492b085b775cfa1cfd"} Nov 03 17:26:09.476661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.476629 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:09.581363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.581210 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\") pod \"d1190400-8d4a-4943-acd3-6d56fd59d03d\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " Nov 03 17:26:09.581842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.581811 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\") pod \"d1190400-8d4a-4943-acd3-6d56fd59d03d\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " Nov 03 17:26:09.581975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.581905 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\" (UniqueName: \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\") pod \"d1190400-8d4a-4943-acd3-6d56fd59d03d\" (UID: \"d1190400-8d4a-4943-acd3-6d56fd59d03d\") " Nov 03 17:26:09.582906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.582883 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca" pod "d1190400-8d4a-4943-acd3-6d56fd59d03d" (UID: "d1190400-8d4a-4943-acd3-6d56fd59d03d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:09.585565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.585533 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca" pod "d1190400-8d4a-4943-acd3-6d56fd59d03d" (UID: "d1190400-8d4a-4943-acd3-6d56fd59d03d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:09.586044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.586009 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624" pod "d1190400-8d4a-4943-acd3-6d56fd59d03d" (UID: "d1190400-8d4a-4943-acd3-6d56fd59d03d"). InnerVolumeSpecName "kube-api-access-cp624". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:09.684433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.684398 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.684433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.684428 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1190400-8d4a-4943-acd3-6d56fd59d03d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.684668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.684440 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp624\" (UniqueName: \"kubernetes.io/projected/d1190400-8d4a-4943-acd3-6d56fd59d03d-kube-api-access-cp624\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.746158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.746125 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" Nov 03 17:26:09.746439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.746164 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84" event={"ID":"d1190400-8d4a-4943-acd3-6d56fd59d03d","Type":"ContainerDied","Data":"0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe"} Nov 03 17:26:09.746439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:09.746222 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fb57f970a425a9328cbf2c8f9f4f854060021d224050d45ce0e0897a936befe" Nov 03 17:26:11.873867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:11.873831 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:26:11.874425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:11.874153 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" containerID="cri-o://ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90" gracePeriod=30 Nov 03 17:26:17.391267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.391232 6001 patch_prober.go:29] interesting pod/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" start-of-body= Nov 03 17:26:17.391744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.391297 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.94:15021/healthz/ready\": dial tcp 10.134.1.94:15021: connect: connection refused" Nov 03 17:26:17.413122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.413079 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj"] Nov 03 17:26:17.415156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415123 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:17.415156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415154 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:17.415378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415244 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d1190400-8d4a-4943-acd3-6d56fd59d03d" containerName="revoke-keys" Nov 03 17:26:17.415378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415251 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1190400-8d4a-4943-acd3-6d56fd59d03d" containerName="revoke-keys" Nov 03 17:26:17.415725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415702 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1190400-8d4a-4943-acd3-6d56fd59d03d" containerName="revoke-keys" Nov 03 17:26:17.415832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.415760 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3529aefc-dbb5-415b-9192-b233f5cf4cba" containerName="istio-proxy" Nov 03 17:26:17.422791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.422085 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.438414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.438383 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj"] Nov 03 17:26:17.441391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.441364 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:26:17.441543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.441401 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:26:17.445497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.442905 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:26:17.445497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.444497 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:26:17.469792 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:26:17.469761 6001 logging.go:55] [core] [Channel #23487 SubChannel #23488]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.105:50052", ServerName: "10.134.1.105:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.105:50052: connect: connection refused" Nov 03 17:26:17.555068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.555023 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.555347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.555254 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.555347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.555303 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd7kn\" (UniqueName: \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.656549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.656511 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd7kn\" (UniqueName: \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.658299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.656766 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.658299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.656901 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.658299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.657777 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.661135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.660227 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.681646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.681614 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd7kn\" (UniqueName: \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\") pod \"maas-api-revoke-sub-d0d7a7ab-2qvxj\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.687048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.687021 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:26:17.689735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.689709 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:26:17.742676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.742615 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:17.759084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759046 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\" (UniqueName: \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\") pod \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " Nov 03 17:26:17.759303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759121 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759159 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759191 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759269 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759349 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759383 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759512 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.759614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759561 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\") pod \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\" (UID: \"078a5cb0-78be-4650-9cdb-1dbaf17d2c67\") " Nov 03 17:26:17.759831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.759643 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.760194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.760167 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.760570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.760546 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.760767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.760741 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume" pod "078a5cb0-78be-4650-9cdb-1dbaf17d2c67" (UID: "078a5cb0-78be-4650-9cdb-1dbaf17d2c67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:17.760852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.760342 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.761107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.761086 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.761405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.761381 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:17.773930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.773866 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.773930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.773902 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq" pod "078a5cb0-78be-4650-9cdb-1dbaf17d2c67" (UID: "078a5cb0-78be-4650-9cdb-1dbaf17d2c67"). InnerVolumeSpecName "kube-api-access-q8nsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.778694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.774219 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:17.778694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.778631 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "kube-api-access-kh44b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.778694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.778670 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.836529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.836496 6001 generic.go:356] "Generic (PLEG): container finished" podID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerID="0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf" exitCode=0 Nov 03 17:26:17.836786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.836561 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" Nov 03 17:26:17.836885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.836557 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" event={"ID":"078a5cb0-78be-4650-9cdb-1dbaf17d2c67","Type":"ContainerDied","Data":"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf"} Nov 03 17:26:17.836946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.836928 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" event={"ID":"078a5cb0-78be-4650-9cdb-1dbaf17d2c67","Type":"ContainerDied","Data":"e3a3c8ba3195770be4dbefc87d1f90fbe0ad347da417958a94639a9950d70368"} Nov 03 17:26:17.836995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.836963 6001 scope.go:122] "RemoveContainer" containerID="0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf" Nov 03 17:26:17.841679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.840876 6001 generic.go:356] "Generic (PLEG): container finished" podID="03a1743a-fe09-4413-87db-42f5e11da91c" containerID="ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90" exitCode=0 Nov 03 17:26:17.841679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.841113 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerDied","Data":"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90"} Nov 03 17:26:17.841679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.841160 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" event={"ID":"03a1743a-fe09-4413-87db-42f5e11da91c","Type":"ContainerDied","Data":"c4205fa6fe670d68345325fd327bba99fbe462020f93740c45700cc918e25f9e"} Nov 03 17:26:17.841679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.841237 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.861484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\") pod \"03a1743a-fe09-4413-87db-42f5e11da91c\" (UID: \"03a1743a-fe09-4413-87db-42f5e11da91c\") " Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.861722 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket" pod "03a1743a-fe09-4413-87db-42f5e11da91c" (UID: "03a1743a-fe09-4413-87db-42f5e11da91c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862196 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862214 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862228 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862239 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862250 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8nsq\" (UniqueName: \"kubernetes.io/projected/078a5cb0-78be-4650-9cdb-1dbaf17d2c67-kube-api-access-q8nsq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862261 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03a1743a-fe09-4413-87db-42f5e11da91c-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862275 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862286 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh44b\" (UniqueName: \"kubernetes.io/projected/03a1743a-fe09-4413-87db-42f5e11da91c-kube-api-access-kh44b\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862298 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03a1743a-fe09-4413-87db-42f5e11da91c-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.862988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.862310 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03a1743a-fe09-4413-87db-42f5e11da91c-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.879246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.877793 6001 scope.go:122] "RemoveContainer" containerID="0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf" Nov 03 17:26:17.879246 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:17.878218 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf\": container with ID starting with 0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf not found: ID does not exist" containerID="0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf" Nov 03 17:26:17.879246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.878246 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf"} err="failed to get container status \"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf\": rpc error: code = NotFound desc = could not find container \"0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf\": container with ID starting with 0028395d6a81c621e88dd92e22a7ddb85a69c4c45f5eeff3bdd89110b5f58caf not found: ID does not exist" Nov 03 17:26:17.879246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.878266 6001 scope.go:122] "RemoveContainer" containerID="ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90" Nov 03 17:26:17.892607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.892518 6001 scope.go:122] "RemoveContainer" containerID="e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697" Nov 03 17:26:17.906736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.906666 6001 scope.go:122] "RemoveContainer" containerID="ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90" Nov 03 17:26:17.911741 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:17.911679 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90\": container with ID starting with ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90 not found: ID does not exist" containerID="ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90" Nov 03 17:26:17.912059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.911722 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90"} err="failed to get container status \"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90\": rpc error: code = NotFound desc = could not find container \"ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90\": container with ID starting with ad65ab42191678d6ec7d30e54685ca34d99454724d89c23503926b0df425da90 not found: ID does not exist" Nov 03 17:26:17.912059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.911996 6001 scope.go:122] "RemoveContainer" containerID="e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697" Nov 03 17:26:17.914459 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:17.914331 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697\": container with ID starting with e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697 not found: ID does not exist" containerID="e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697" Nov 03 17:26:17.914459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.914364 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697"} err="failed to get container status \"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697\": rpc error: code = NotFound desc = could not find container \"e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697\": container with ID starting with e1d161077727075fe2c6b451e52b6c06344cbc88843030dd7aae1705bdaad697 not found: ID does not exist" Nov 03 17:26:17.918797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.918738 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:26:17.924376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:17.924351 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h"] Nov 03 17:26:18.080594 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:26:18.080564 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50662027_ba44_4166_9727_cc5ced001756.slice/crio-1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209 WatchSource:0}: Error finding container 1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209: Status 404 returned error can't find the container with id 1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209 Nov 03 17:26:18.082110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.082082 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj"] Nov 03 17:26:18.204073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.204038 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:26:18.208502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.208467 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-527051-openshift-default-bfb8678d5-mj4cd"] Nov 03 17:26:18.469054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.468953 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-547925-698c7697bc-7sh6h" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.105:50052\" within 1s: context deadline exceeded" Nov 03 17:26:18.793828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.793794 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" path="/var/lib/kubelet/pods/03a1743a-fe09-4413-87db-42f5e11da91c/volumes" Nov 03 17:26:18.794321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.794307 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" path="/var/lib/kubelet/pods/078a5cb0-78be-4650-9cdb-1dbaf17d2c67/volumes" Nov 03 17:26:18.870185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.870147 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" event={"ID":"50662027-ba44-4166-9727-cc5ced001756","Type":"ContainerStarted","Data":"ee60c851dfc48497eff2a5fb4b7dd2b5fae1975cf98576a0642ef3200b4d7d64"} Nov 03 17:26:18.870399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.870220 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" event={"ID":"50662027-ba44-4166-9727-cc5ced001756","Type":"ContainerStarted","Data":"1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209"} Nov 03 17:26:18.915155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:18.915102 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" podStartSLOduration=1.915083992 podStartE2EDuration="1.915083992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:26:18.912960565 +0000 UTC m=+4274.246174134" watchObservedRunningTime="2026-11-03 17:26:18.915083992 +0000 UTC m=+4274.248297562" Nov 03 17:26:19.883116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:19.883080 6001 generic.go:356] "Generic (PLEG): container finished" podID="50662027-ba44-4166-9727-cc5ced001756" containerID="ee60c851dfc48497eff2a5fb4b7dd2b5fae1975cf98576a0642ef3200b4d7d64" exitCode=0 Nov 03 17:26:19.883596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:19.883162 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" event={"ID":"50662027-ba44-4166-9727-cc5ced001756","Type":"ContainerDied","Data":"ee60c851dfc48497eff2a5fb4b7dd2b5fae1975cf98576a0642ef3200b4d7d64"} Nov 03 17:26:21.064712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.064686 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:21.130182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.130137 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\" (UniqueName: \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\") pod \"50662027-ba44-4166-9727-cc5ced001756\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " Nov 03 17:26:21.130383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.130295 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\") pod \"50662027-ba44-4166-9727-cc5ced001756\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " Nov 03 17:26:21.130479 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.130409 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\") pod \"50662027-ba44-4166-9727-cc5ced001756\" (UID: \"50662027-ba44-4166-9727-cc5ced001756\") " Nov 03 17:26:21.130927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.130896 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca" pod "50662027-ba44-4166-9727-cc5ced001756" (UID: "50662027-ba44-4166-9727-cc5ced001756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:21.131370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.131345 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50662027-ba44-4166-9727-cc5ced001756-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:21.132739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.132712 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn" pod "50662027-ba44-4166-9727-cc5ced001756" (UID: "50662027-ba44-4166-9727-cc5ced001756"). InnerVolumeSpecName "kube-api-access-rd7kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:21.133018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.132993 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca" pod "50662027-ba44-4166-9727-cc5ced001756" (UID: "50662027-ba44-4166-9727-cc5ced001756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:21.233313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.233276 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50662027-ba44-4166-9727-cc5ced001756-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:21.233313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.233307 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd7kn\" (UniqueName: \"kubernetes.io/projected/50662027-ba44-4166-9727-cc5ced001756-kube-api-access-rd7kn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:21.901937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.901904 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" Nov 03 17:26:21.902128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.901902 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj" event={"ID":"50662027-ba44-4166-9727-cc5ced001756","Type":"ContainerDied","Data":"1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209"} Nov 03 17:26:21.902128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:21.902032 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1491753ed74970ef0068f53b86f1cc04878d23868fb8ab42ef2641ffba499209" Nov 03 17:26:22.252676 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:22.252634 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:26:26.696302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:26.696265 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:26:26.735435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:26.735400 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:26:31.697091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.697042 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" podUID="7869d404-f872-4e91-974c-2ed264c1c0b7" containerName="payload-processing" containerID="cri-o://8442b3ac5fee0cea320d277962183e9255bfd31c5fbfc3e876ab107e7e431c39" gracePeriod=25 Nov 03 17:26:31.736090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.736013 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" containerID="cri-o://9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b" gracePeriod=25 Nov 03 17:26:31.997085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.997034 6001 generic.go:356] "Generic (PLEG): container finished" podID="7869d404-f872-4e91-974c-2ed264c1c0b7" containerID="8442b3ac5fee0cea320d277962183e9255bfd31c5fbfc3e876ab107e7e431c39" exitCode=0 Nov 03 17:26:31.997261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.997098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" event={"ID":"7869d404-f872-4e91-974c-2ed264c1c0b7","Type":"ContainerDied","Data":"8442b3ac5fee0cea320d277962183e9255bfd31c5fbfc3e876ab107e7e431c39"} Nov 03 17:26:31.997261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.997148 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" event={"ID":"7869d404-f872-4e91-974c-2ed264c1c0b7","Type":"ContainerDied","Data":"a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106"} Nov 03 17:26:31.997261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.997164 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f1bc20a27ce4d13e7c24eb48eebe8e825fb022b1a5f5863d03009f27093106" Nov 03 17:26:31.999248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:31.999231 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:26:32.123829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.123792 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\") pod \"7869d404-f872-4e91-974c-2ed264c1c0b7\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " Nov 03 17:26:32.124031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.123902 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\" (UniqueName: \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\") pod \"7869d404-f872-4e91-974c-2ed264c1c0b7\" (UID: \"7869d404-f872-4e91-974c-2ed264c1c0b7\") " Nov 03 17:26:32.124182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.124158 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume" pod "7869d404-f872-4e91-974c-2ed264c1c0b7" (UID: "7869d404-f872-4e91-974c-2ed264c1c0b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:32.124552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.124536 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7869d404-f872-4e91-974c-2ed264c1c0b7-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:32.126139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.126116 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh" pod "7869d404-f872-4e91-974c-2ed264c1c0b7" (UID: "7869d404-f872-4e91-974c-2ed264c1c0b7"). InnerVolumeSpecName "kube-api-access-5dpqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:32.226239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:32.226203 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dpqh\" (UniqueName: \"kubernetes.io/projected/7869d404-f872-4e91-974c-2ed264c1c0b7-kube-api-access-5dpqh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:33.005415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:33.005359 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g" Nov 03 17:26:33.063597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:33.063506 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:26:33.078196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:33.078160 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-236539-5655bf77f-kkp6g"] Nov 03 17:26:33.804382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:33.804339 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:34.792134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:34.792099 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7869d404-f872-4e91-974c-2ed264c1c0b7" path="/var/lib/kubelet/pods/7869d404-f872-4e91-974c-2ed264c1c0b7/volumes" Nov 03 17:26:43.803948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:43.803902 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:44.759384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:44.759343 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:26:44.759644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:44.759562 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" containerID="cri-o://12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e" gracePeriod=30 Nov 03 17:26:50.073938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.073905 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:26:50.172612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.172563 6001 generic.go:356] "Generic (PLEG): container finished" podID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerID="12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e" exitCode=0 Nov 03 17:26:50.172923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.172900 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" Nov 03 17:26:50.173048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.172890 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" event={"ID":"bdd79931-579e-4fb5-8a58-6819302d32cb","Type":"ContainerDied","Data":"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e"} Nov 03 17:26:50.173097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.173047 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb" event={"ID":"bdd79931-579e-4fb5-8a58-6819302d32cb","Type":"ContainerDied","Data":"4d5ae7ba0c09e8f860b346bd03075dca6a4fd81c7510cf01168136ba5817433f"} Nov 03 17:26:50.173097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.173088 6001 scope.go:122] "RemoveContainer" containerID="12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e" Nov 03 17:26:50.175718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.175699 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.175816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.175771 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.175816 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.175801 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.175976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.175959 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.176070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176052 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.176127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176091 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.176224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176214 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.176278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176249 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.176350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176327 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\") pod \"bdd79931-579e-4fb5-8a58-6819302d32cb\" (UID: \"bdd79931-579e-4fb5-8a58-6819302d32cb\") " Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176541 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176590 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.176782 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.177204 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.177228 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.177247 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.177480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.177422 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:50.178104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.178048 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.178662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.178636 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:50.179161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.179137 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "kube-api-access-xm5f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:50.179869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.179838 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:50.183060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.183034 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token" pod "bdd79931-579e-4fb5-8a58-6819302d32cb" (UID: "bdd79931-579e-4fb5-8a58-6819302d32cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:50.225520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.225496 6001 scope.go:122] "RemoveContainer" containerID="12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e" Nov 03 17:26:50.225920 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:50.225892 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e\": container with ID starting with 12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e not found: ID does not exist" containerID="12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e" Nov 03 17:26:50.225975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.225931 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e"} err="failed to get container status \"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e\": rpc error: code = NotFound desc = could not find container \"12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e\": container with ID starting with 12f21e06337f676d83ceb6f70f6966583bd70c517f959de10804aa2f7769a11e not found: ID does not exist" Nov 03 17:26:50.279162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279121 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm5f8\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-kube-api-access-xm5f8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.279162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279156 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.279162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279167 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.279426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279178 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.279426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279186 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd79931-579e-4fb5-8a58-6819302d32cb-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.279426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.279195 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd79931-579e-4fb5-8a58-6819302d32cb-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:50.537627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.537584 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:26:50.550088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.550052 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-547925-openshift-default-5f58b679d9-6vcpb"] Nov 03 17:26:50.793008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:50.792905 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" path="/var/lib/kubelet/pods/bdd79931-579e-4fb5-8a58-6819302d32cb/volumes" Nov 03 17:26:51.938627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:51.938591 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:26:52.040605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.018163 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\") pod \"17883199-964b-4c64-a594-246fde087c55\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " Nov 03 17:26:52.040605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.018221 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\" (UniqueName: \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\") pod \"17883199-964b-4c64-a594-246fde087c55\" (UID: \"17883199-964b-4c64-a594-246fde087c55\") " Nov 03 17:26:52.040605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.021829 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume" pod "17883199-964b-4c64-a594-246fde087c55" (UID: "17883199-964b-4c64-a594-246fde087c55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:52.047145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.047101 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2" pod "17883199-964b-4c64-a594-246fde087c55" (UID: "17883199-964b-4c64-a594-246fde087c55"). InnerVolumeSpecName "kube-api-access-dh9m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:52.121145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.121101 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17883199-964b-4c64-a594-246fde087c55-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.121145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.121136 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9m2\" (UniqueName: \"kubernetes.io/projected/17883199-964b-4c64-a594-246fde087c55-kube-api-access-dh9m2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.202751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.202713 6001 generic.go:356] "Generic (PLEG): container finished" podID="17883199-964b-4c64-a594-246fde087c55" containerID="9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b" exitCode=0 Nov 03 17:26:52.202953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.202744 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" event={"ID":"17883199-964b-4c64-a594-246fde087c55","Type":"ContainerDied","Data":"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b"} Nov 03 17:26:52.202953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.202765 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" Nov 03 17:26:52.202953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.202808 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2" event={"ID":"17883199-964b-4c64-a594-246fde087c55","Type":"ContainerDied","Data":"9285ac63aba881761f25d9c80190b673afac68bc62b412d6f2ffc430b9e6e5df"} Nov 03 17:26:52.202953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.202831 6001 scope.go:122] "RemoveContainer" containerID="9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b" Nov 03 17:26:52.215382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.215363 6001 scope.go:122] "RemoveContainer" containerID="9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b" Nov 03 17:26:52.215736 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:52.215716 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b\": container with ID starting with 9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b not found: ID does not exist" containerID="9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b" Nov 03 17:26:52.215806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.215748 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b"} err="failed to get container status \"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b\": rpc error: code = NotFound desc = could not find container \"9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b\": container with ID starting with 9b7439e4da0c0a42fcfa9de219fc69aa675c612dbdd09baf68a7d3b1fee9b70b not found: ID does not exist" Nov 03 17:26:52.251255 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:52.251220 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:26:52.264892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.264858 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:26:52.269122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.269097 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-236539-f47584c87-qpbp2"] Nov 03 17:26:52.605474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.605429 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:26:52.605723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.605698 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" containerID="cri-o://f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e" gracePeriod=30 Nov 03 17:26:52.823055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:52.823015 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17883199-964b-4c64-a594-246fde087c55" path="/var/lib/kubelet/pods/17883199-964b-4c64-a594-246fde087c55/volumes" Nov 03 17:26:57.169204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:57.169168 6001 patch_prober.go:29] interesting pod/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:26:57.169665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:57.169227 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:26:57.909569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:57.909539 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:26:58.023218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023320 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023463 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023536 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023562 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023593 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023661 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023699 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.023821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.023801 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\") pod \"37786abd-37ec-4fda-8e9b-a14e97a0711c\" (UID: \"37786abd-37ec-4fda-8e9b-a14e97a0711c\") " Nov 03 17:26:58.024166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024127 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:58.024506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024480 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:58.024679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024655 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:58.024818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024790 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:58.024904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024836 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:58.024904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024869 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.025034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.024908 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.026265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.026203 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:58.027077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.027051 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:58.027165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.027130 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:58.027485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.027433 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n" pod "37786abd-37ec-4fda-8e9b-a14e97a0711c" (UID: "37786abd-37ec-4fda-8e9b-a14e97a0711c"). InnerVolumeSpecName "kube-api-access-stp6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126622 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126662 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126671 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126680 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stp6n\" (UniqueName: \"kubernetes.io/projected/37786abd-37ec-4fda-8e9b-a14e97a0711c-kube-api-access-stp6n\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126690 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37786abd-37ec-4fda-8e9b-a14e97a0711c-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126699 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37786abd-37ec-4fda-8e9b-a14e97a0711c-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.126723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.126707 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37786abd-37ec-4fda-8e9b-a14e97a0711c-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:26:58.257942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.257905 6001 generic.go:356] "Generic (PLEG): container finished" podID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerID="f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e" exitCode=0 Nov 03 17:26:58.258423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.257961 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" Nov 03 17:26:58.258423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.257979 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerDied","Data":"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e"} Nov 03 17:26:58.258423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.258024 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8" event={"ID":"37786abd-37ec-4fda-8e9b-a14e97a0711c","Type":"ContainerDied","Data":"b455c7a7332a2586b9d7c7a72d1aebc7720892ed4b30b2caf75fba86b56f1dac"} Nov 03 17:26:58.258423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.258044 6001 scope.go:122] "RemoveContainer" containerID="f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e" Nov 03 17:26:58.270790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.270769 6001 scope.go:122] "RemoveContainer" containerID="1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591" Nov 03 17:26:58.282761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.282735 6001 scope.go:122] "RemoveContainer" containerID="f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e" Nov 03 17:26:58.283116 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:58.283095 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e\": container with ID starting with f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e not found: ID does not exist" containerID="f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e" Nov 03 17:26:58.283170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.283126 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e"} err="failed to get container status \"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e\": rpc error: code = NotFound desc = could not find container \"f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e\": container with ID starting with f2e7500a4fb7420c68d73fd94ca7e15f392c799a15d8f6fcdc98352de2d0a00e not found: ID does not exist" Nov 03 17:26:58.283170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.283145 6001 scope.go:122] "RemoveContainer" containerID="1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591" Nov 03 17:26:58.283434 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:26:58.283414 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591\": container with ID starting with 1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591 not found: ID does not exist" containerID="1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591" Nov 03 17:26:58.283496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.283478 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591"} err="failed to get container status \"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591\": rpc error: code = NotFound desc = could not find container \"1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591\": container with ID starting with 1588ac379443044ef9d71d86952bdba6d952f7eafae2fef42786acc07a0ef591 not found: ID does not exist" Nov 03 17:26:58.316154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.316113 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:26:58.329185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.329151 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-472bee-openshift-default-7dd789f88f-twpp8"] Nov 03 17:26:58.790980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:58.790945 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" path="/var/lib/kubelet/pods/37786abd-37ec-4fda-8e9b-a14e97a0711c/volumes" Nov 03 17:26:59.367316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:59.367280 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:26:59.405643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:26:59.405605 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:27:04.368212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.368157 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" podUID="954947d0-810d-49f4-b404-f6d36f01ecb9" containerName="payload-processing" containerID="cri-o://29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f" gracePeriod=25 Nov 03 17:27:04.406695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.406642 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" containerID="cri-o://399f977876b3440394bd2057319c7ecb5a5ef565036b89d35e09446b9e9e66fa" gracePeriod=25 Nov 03 17:27:04.686414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.686386 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:27:04.837045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.837003 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\" (UniqueName: \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\") pod \"954947d0-810d-49f4-b404-f6d36f01ecb9\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " Nov 03 17:27:04.837236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.837213 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\") pod \"954947d0-810d-49f4-b404-f6d36f01ecb9\" (UID: \"954947d0-810d-49f4-b404-f6d36f01ecb9\") " Nov 03 17:27:04.837639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.837616 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume" pod "954947d0-810d-49f4-b404-f6d36f01ecb9" (UID: "954947d0-810d-49f4-b404-f6d36f01ecb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:04.837836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.837818 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954947d0-810d-49f4-b404-f6d36f01ecb9-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:04.839384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.839363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx" pod "954947d0-810d-49f4-b404-f6d36f01ecb9" (UID: "954947d0-810d-49f4-b404-f6d36f01ecb9"). InnerVolumeSpecName "kube-api-access-vvmvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:04.939706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:04.939624 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvmvx\" (UniqueName: \"kubernetes.io/projected/954947d0-810d-49f4-b404-f6d36f01ecb9-kube-api-access-vvmvx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:05.319931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.319897 6001 generic.go:356] "Generic (PLEG): container finished" podID="954947d0-810d-49f4-b404-f6d36f01ecb9" containerID="29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f" exitCode=0 Nov 03 17:27:05.320123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.319940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" event={"ID":"954947d0-810d-49f4-b404-f6d36f01ecb9","Type":"ContainerDied","Data":"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f"} Nov 03 17:27:05.320123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.319953 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" Nov 03 17:27:05.320123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.319988 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7" event={"ID":"954947d0-810d-49f4-b404-f6d36f01ecb9","Type":"ContainerDied","Data":"99c44d2988b496df6c49707ef2480f01f8209639abdf7e83a6863b93c8f08699"} Nov 03 17:27:05.320123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.320012 6001 scope.go:122] "RemoveContainer" containerID="29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f" Nov 03 17:27:05.332638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.332603 6001 scope.go:122] "RemoveContainer" containerID="29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f" Nov 03 17:27:05.332986 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:05.332965 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f\": container with ID starting with 29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f not found: ID does not exist" containerID="29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f" Nov 03 17:27:05.333052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.332994 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f"} err="failed to get container status \"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f\": rpc error: code = NotFound desc = could not find container \"29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f\": container with ID starting with 29498434bb9a0b5a053dcac2fcf9682ad9bdc8429f31df8ad38301498fcec55f not found: ID does not exist" Nov 03 17:27:05.377010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.376968 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:27:05.389139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:05.389104 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-89df9a-6dcc4c8d8f-7php7"] Nov 03 17:27:06.791722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:06.791683 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954947d0-810d-49f4-b404-f6d36f01ecb9" path="/var/lib/kubelet/pods/954947d0-810d-49f4-b404-f6d36f01ecb9/volumes" Nov 03 17:27:08.593074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:08.593028 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:27:11.430006 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:11.429938 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:27:11.431074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:11.430219 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" containerID="cri-o://1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde" gracePeriod=30 Nov 03 17:27:13.164768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:13.164682 6001 scope.go:122] "RemoveContainer" containerID="8442b3ac5fee0cea320d277962183e9255bfd31c5fbfc3e876ab107e7e431c39" Nov 03 17:27:16.513538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.513497 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw"] Nov 03 17:27:16.514349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514332 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7869d404-f872-4e91-974c-2ed264c1c0b7" containerName="payload-processing" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514350 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7869d404-f872-4e91-974c-2ed264c1c0b7" containerName="payload-processing" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514371 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514376 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514385 6001 cpu_manager.go:414] "containerMap: removing container" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514390 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:16.514399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514398 6001 cpu_manager.go:414] "containerMap: removing container" podUID="954947d0-810d-49f4-b404-f6d36f01ecb9" containerName="payload-processing" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514403 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954947d0-810d-49f4-b404-f6d36f01ecb9" containerName="payload-processing" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514419 6001 cpu_manager.go:414] "containerMap: removing container" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514424 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514456 6001 cpu_manager.go:414] "containerMap: removing container" podUID="50662027-ba44-4166-9727-cc5ced001756" containerName="revoke-keys" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514464 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50662027-ba44-4166-9727-cc5ced001756" containerName="revoke-keys" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514492 6001 cpu_manager.go:414] "containerMap: removing container" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514499 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514514 6001 cpu_manager.go:414] "containerMap: removing container" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514522 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514537 6001 cpu_manager.go:414] "containerMap: removing container" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" Nov 03 17:27:16.514636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514542 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514681 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd79931-579e-4fb5-8a58-6819302d32cb" containerName="istio-proxy" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514691 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514698 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="954947d0-810d-49f4-b404-f6d36f01ecb9" containerName="payload-processing" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514707 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="17883199-964b-4c64-a594-246fde087c55" containerName="payload-pre-processing" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514714 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="50662027-ba44-4166-9727-cc5ced001756" containerName="revoke-keys" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514723 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514730 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="37786abd-37ec-4fda-8e9b-a14e97a0711c" containerName="istio-proxy" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514737 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="078a5cb0-78be-4650-9cdb-1dbaf17d2c67" containerName="payload-processing" Nov 03 17:27:16.514984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.514747 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7869d404-f872-4e91-974c-2ed264c1c0b7" containerName="payload-processing" Nov 03 17:27:16.518655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.518631 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.538673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.538639 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw"] Nov 03 17:27:16.545831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.545799 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:27:16.546020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.545943 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:27:16.547704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.547678 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:27:16.547819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.547751 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:27:16.577548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.577513 6001 patch_prober.go:29] interesting pod/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" start-of-body= Nov 03 17:27:16.577704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.577576 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.98:15021/healthz/ready\": dial tcp 10.134.1.98:15021: connect: connection refused" Nov 03 17:27:16.633285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.632751 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.633285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.633028 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.633285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.633123 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkh9\" (UniqueName: \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.734555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.734518 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.734715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.734669 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.734715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.734694 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkh9\" (UniqueName: \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.735343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.735312 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.737124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.737097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.759940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.759907 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkh9\" (UniqueName: \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\") pod \"maas-api-revoke-sub-5110bcd2-w8wvw\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.760170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.760149 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:27:16.840357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.840321 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.840892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.840868 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841510 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841576 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841603 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841722 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841764 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841796 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.841981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841874 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\") pod \"863f8808-b938-4728-86f4-ec5ad8c7758a\" (UID: \"863f8808-b938-4728-86f4-ec5ad8c7758a\") " Nov 03 17:27:16.843405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.841407 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:16.843405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.842286 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:16.843405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.842578 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:16.843405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.842910 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:16.843405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.843036 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "kube-api-access-b7g49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:16.844024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.844003 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:16.845233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.845211 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:16.845838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.845813 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:16.846053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.846022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:16.847130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847105 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847133 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847147 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847162 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g49\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-kube-api-access-b7g49\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847176 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/863f8808-b938-4728-86f4-ec5ad8c7758a-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847341 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847360 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847372 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:16.847549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.847422 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy" pod "863f8808-b938-4728-86f4-ec5ad8c7758a" (UID: "863f8808-b938-4728-86f4-ec5ad8c7758a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:16.948666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:16.948632 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/863f8808-b938-4728-86f4-ec5ad8c7758a-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:17.159223 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:27:17.159182 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a86608_4201_4815_b4c8_dff750c2e0d1.slice/crio-302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6 WatchSource:0}: Error finding container 302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6: Status 404 returned error can't find the container with id 302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6 Nov 03 17:27:17.160898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.160765 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw"] Nov 03 17:27:17.479518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.479486 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" event={"ID":"c2a86608-4201-4815-b4c8-dff750c2e0d1","Type":"ContainerStarted","Data":"302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6"} Nov 03 17:27:17.481610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.481584 6001 generic.go:356] "Generic (PLEG): container finished" podID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerID="1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde" exitCode=0 Nov 03 17:27:17.481789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.481631 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" Nov 03 17:27:17.481789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.481656 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerDied","Data":"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde"} Nov 03 17:27:17.481789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.481718 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4" event={"ID":"863f8808-b938-4728-86f4-ec5ad8c7758a","Type":"ContainerDied","Data":"9bab48ca14e8e6575a37c34427f10d6bdc899a5a64295c60207b4d901524a3ad"} Nov 03 17:27:17.481789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.481750 6001 scope.go:122] "RemoveContainer" containerID="1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde" Nov 03 17:27:17.493509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.493487 6001 scope.go:122] "RemoveContainer" containerID="c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c" Nov 03 17:27:17.505053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.505029 6001 scope.go:122] "RemoveContainer" containerID="1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde" Nov 03 17:27:17.505382 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:17.505360 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde\": container with ID starting with 1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde not found: ID does not exist" containerID="1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde" Nov 03 17:27:17.505463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.505390 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde"} err="failed to get container status \"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde\": rpc error: code = NotFound desc = could not find container \"1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde\": container with ID starting with 1acf332f2ec55febbcc6968de9897f2677d3f950ac2091cf93d4cbaa6f504dde not found: ID does not exist" Nov 03 17:27:17.505463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.505408 6001 scope.go:122] "RemoveContainer" containerID="c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c" Nov 03 17:27:17.505759 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:17.505739 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c\": container with ID starting with c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c not found: ID does not exist" containerID="c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c" Nov 03 17:27:17.505828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.505765 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c"} err="failed to get container status \"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c\": rpc error: code = NotFound desc = could not find container \"c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c\": container with ID starting with c8403c6f46cb8642aaffbee78bd96b480b56fcbe80c69ecb7a6b4c307778989c not found: ID does not exist" Nov 03 17:27:17.538896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.538860 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:27:17.552044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:17.552014 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-236539-openshift-default-789fd85b45-fs7k4"] Nov 03 17:27:18.545378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:18.545285 6001 generic.go:356] "Generic (PLEG): container finished" podID="c2a86608-4201-4815-b4c8-dff750c2e0d1" containerID="54cbecd24499dc5a679951fcdbb47d626121692266cf1c23694806d97549ae51" exitCode=0 Nov 03 17:27:18.545378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:18.545361 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" event={"ID":"c2a86608-4201-4815-b4c8-dff750c2e0d1","Type":"ContainerDied","Data":"54cbecd24499dc5a679951fcdbb47d626121692266cf1c23694806d97549ae51"} Nov 03 17:27:18.593033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:18.592993 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:27:18.792386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:18.792351 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" path="/var/lib/kubelet/pods/863f8808-b938-4728-86f4-ec5ad8c7758a/volumes" Nov 03 17:27:19.728205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.728175 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:19.792711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.792672 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\") pod \"c2a86608-4201-4815-b4c8-dff750c2e0d1\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " Nov 03 17:27:19.792941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.792818 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\" (UniqueName: \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\") pod \"c2a86608-4201-4815-b4c8-dff750c2e0d1\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " Nov 03 17:27:19.792941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.792909 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\") pod \"c2a86608-4201-4815-b4c8-dff750c2e0d1\" (UID: \"c2a86608-4201-4815-b4c8-dff750c2e0d1\") " Nov 03 17:27:19.793173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.793141 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca" pod "c2a86608-4201-4815-b4c8-dff750c2e0d1" (UID: "c2a86608-4201-4815-b4c8-dff750c2e0d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:19.793754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.793728 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:19.795547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.795521 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca" pod "c2a86608-4201-4815-b4c8-dff750c2e0d1" (UID: "c2a86608-4201-4815-b4c8-dff750c2e0d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:19.796160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.795622 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9" pod "c2a86608-4201-4815-b4c8-dff750c2e0d1" (UID: "c2a86608-4201-4815-b4c8-dff750c2e0d1"). InnerVolumeSpecName "kube-api-access-pkkh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:19.895645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.895603 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkh9\" (UniqueName: \"kubernetes.io/projected/c2a86608-4201-4815-b4c8-dff750c2e0d1-kube-api-access-pkkh9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:19.895645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:19.895644 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2a86608-4201-4815-b4c8-dff750c2e0d1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:20.562179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:20.562137 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" event={"ID":"c2a86608-4201-4815-b4c8-dff750c2e0d1","Type":"ContainerDied","Data":"302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6"} Nov 03 17:27:20.562179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:20.562157 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw" Nov 03 17:27:20.562414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:20.562189 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="302b9ed6276c5d799f4ae7f6774e3c6189a76f4eaabccb254d7881def90bcab6" Nov 03 17:27:22.253735 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:22.253697 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:27:24.637161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.637120 6001 generic.go:356] "Generic (PLEG): container finished" podID="1d7990af-35df-4fee-b294-a20f641d6742" containerID="399f977876b3440394bd2057319c7ecb5a5ef565036b89d35e09446b9e9e66fa" exitCode=0 Nov 03 17:27:24.637611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.637190 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" event={"ID":"1d7990af-35df-4fee-b294-a20f641d6742","Type":"ContainerDied","Data":"399f977876b3440394bd2057319c7ecb5a5ef565036b89d35e09446b9e9e66fa"} Nov 03 17:27:24.709923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.709892 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:27:24.772745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.772711 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\" (UniqueName: \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\") pod \"1d7990af-35df-4fee-b294-a20f641d6742\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " Nov 03 17:27:24.772967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.772820 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\") pod \"1d7990af-35df-4fee-b294-a20f641d6742\" (UID: \"1d7990af-35df-4fee-b294-a20f641d6742\") " Nov 03 17:27:24.773528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.773500 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume" pod "1d7990af-35df-4fee-b294-a20f641d6742" (UID: "1d7990af-35df-4fee-b294-a20f641d6742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:24.775841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.775813 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8" pod "1d7990af-35df-4fee-b294-a20f641d6742" (UID: "1d7990af-35df-4fee-b294-a20f641d6742"). InnerVolumeSpecName "kube-api-access-fg2h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:24.874914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.874877 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2h8\" (UniqueName: \"kubernetes.io/projected/1d7990af-35df-4fee-b294-a20f641d6742-kube-api-access-fg2h8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:24.874914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:24.874918 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d7990af-35df-4fee-b294-a20f641d6742-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:25.647887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:25.647851 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" Nov 03 17:27:25.647887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:25.647870 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh" event={"ID":"1d7990af-35df-4fee-b294-a20f641d6742","Type":"ContainerDied","Data":"effea5aae53180d21039f7b2a9eb4623c7d9604145faa6a3172958e938d45613"} Nov 03 17:27:25.648438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:25.647940 6001 scope.go:122] "RemoveContainer" containerID="399f977876b3440394bd2057319c7ecb5a5ef565036b89d35e09446b9e9e66fa" Nov 03 17:27:25.703460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:25.703401 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:27:25.716053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:25.716020 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-89df9a-6c4fcc88ff-zlksh"] Nov 03 17:27:26.791127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:26.791086 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7990af-35df-4fee-b294-a20f641d6742" path="/var/lib/kubelet/pods/1d7990af-35df-4fee-b294-a20f641d6742/volumes" Nov 03 17:27:33.979392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.979351 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:27:33.980199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980182 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c2a86608-4201-4815-b4c8-dff750c2e0d1" containerName="revoke-keys" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980201 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a86608-4201-4815-b4c8-dff750c2e0d1" containerName="revoke-keys" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980212 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980217 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980234 6001 cpu_manager.go:414] "containerMap: removing container" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980240 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:33.980257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980255 6001 cpu_manager.go:414] "containerMap: removing container" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980260 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980267 6001 cpu_manager.go:414] "containerMap: removing container" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980272 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980393 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a86608-4201-4815-b4c8-dff750c2e0d1" containerName="revoke-keys" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980405 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7990af-35df-4fee-b294-a20f641d6742" containerName="payload-pre-processing" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980412 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:33.980470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.980418 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="03a1743a-fe09-4413-87db-42f5e11da91c" containerName="istio-proxy" Nov 03 17:27:33.985890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:33.985867 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.011619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.011587 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:27:34.036814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.036777 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.036850 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lpq\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.036918 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.036963 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.037131 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.037181 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.037259 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.037317 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.037523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.037400 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.138969 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139060 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139152 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139197 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lpq\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139231 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139327 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139379 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139415 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139655 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139744 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139830 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.139920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.139861 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.140075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.140058 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.141690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.141672 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.141844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.141824 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.166060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.166023 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.166211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.166177 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lpq\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-2h7hw\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.303802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.303712 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:34.654781 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:27:34.654754 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1526dc29_4450_4968_8976_d965f11b1dd7.slice/crio-2e486ce74a24fbd707e396ad380d9a926c676f2754c59885795f170f4557126e WatchSource:0}: Error finding container 2e486ce74a24fbd707e396ad380d9a926c676f2754c59885795f170f4557126e: Status 404 returned error can't find the container with id 2e486ce74a24fbd707e396ad380d9a926c676f2754c59885795f170f4557126e Nov 03 17:27:34.655629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.655599 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:27:34.657992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.657960 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:27:34.658084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.658027 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:27:34.658084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.658055 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:27:34.730084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:34.730044 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" event={"ID":"1526dc29-4450-4968-8976-d965f11b1dd7","Type":"ContainerStarted","Data":"2e486ce74a24fbd707e396ad380d9a926c676f2754c59885795f170f4557126e"} Nov 03 17:27:35.741846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:35.741801 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" event={"ID":"1526dc29-4450-4968-8976-d965f11b1dd7","Type":"ContainerStarted","Data":"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772"} Nov 03 17:27:35.825679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:35.825623 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podStartSLOduration=2.825609408 podStartE2EDuration="2.825609408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:35.81286355 +0000 UTC m=+4351.146077118" watchObservedRunningTime="2026-11-03 17:27:35.825609408 +0000 UTC m=+4351.158822977" Nov 03 17:27:36.303841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:36.303802 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:36.305934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:36.305905 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:36.306113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:36.305955 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:37.304368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.304323 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:37.304866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.304431 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:37.337880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.337839 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g"] Nov 03 17:27:37.339522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.339492 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="863f8808-b938-4728-86f4-ec5ad8c7758a" containerName="istio-proxy" Nov 03 17:27:37.359806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.359772 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.376888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.376854 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:27:37.377117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.376876 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:27:37.381080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.380251 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g"] Nov 03 17:27:37.403828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.403789 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w49s4\" (UniqueName: \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.404013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.403891 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.404013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.403984 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.429983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.429955 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:27:37.430394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.430375 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:27:37.505526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.505484 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w49s4\" (UniqueName: \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.505526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.505530 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.505817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.505571 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.506285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.506255 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.508140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.508117 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.530335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.530305 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w49s4\" (UniqueName: \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\") pod \"maas-api-revoke-sub-c8abfe23-hd24g\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:37.694920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:37.694840 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:38.025874 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:27:38.025833 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcfc392_e362_4d8a_8f26_f63dbecb07b5.slice/crio-489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368 WatchSource:0}: Error finding container 489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368: Status 404 returned error can't find the container with id 489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368 Nov 03 17:27:38.027877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.027855 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g"] Nov 03 17:27:38.309467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.304868 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:38.309467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.304940 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:38.818611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.818548 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" event={"ID":"6dcfc392-e362-4d8a-8f26-f63dbecb07b5","Type":"ContainerStarted","Data":"df66e99fad6866e20cc40cc1e6cef4eddf744d9bd95e9bbd06cb09846fec103e"} Nov 03 17:27:38.818811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.818636 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" event={"ID":"6dcfc392-e362-4d8a-8f26-f63dbecb07b5","Type":"ContainerStarted","Data":"489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368"} Nov 03 17:27:38.876120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:38.875995 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" podStartSLOduration=1.875974781 podStartE2EDuration="1.875974781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:38.875036595 +0000 UTC m=+4354.208250171" watchObservedRunningTime="2026-11-03 17:27:38.875974781 +0000 UTC m=+4354.209188352" Nov 03 17:27:39.304869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:39.304833 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:39.305047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:39.304909 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:39.828795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:39.828754 6001 generic.go:356] "Generic (PLEG): container finished" podID="6dcfc392-e362-4d8a-8f26-f63dbecb07b5" containerID="df66e99fad6866e20cc40cc1e6cef4eddf744d9bd95e9bbd06cb09846fec103e" exitCode=0 Nov 03 17:27:39.829234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:39.828841 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" event={"ID":"6dcfc392-e362-4d8a-8f26-f63dbecb07b5","Type":"ContainerDied","Data":"df66e99fad6866e20cc40cc1e6cef4eddf744d9bd95e9bbd06cb09846fec103e"} Nov 03 17:27:40.304301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:40.304254 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:40.304559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:40.304315 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:41.058316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.058283 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:41.097988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.097881 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\") pod \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " Nov 03 17:27:41.097988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.097943 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\") pod \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " Nov 03 17:27:41.098287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.098147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\" (UniqueName: \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\") pod \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\" (UID: \"6dcfc392-e362-4d8a-8f26-f63dbecb07b5\") " Nov 03 17:27:41.098287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.098245 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca" pod "6dcfc392-e362-4d8a-8f26-f63dbecb07b5" (UID: "6dcfc392-e362-4d8a-8f26-f63dbecb07b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:41.098889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.098865 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:41.100174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.100142 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca" pod "6dcfc392-e362-4d8a-8f26-f63dbecb07b5" (UID: "6dcfc392-e362-4d8a-8f26-f63dbecb07b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:27:41.100370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.100351 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4" pod "6dcfc392-e362-4d8a-8f26-f63dbecb07b5" (UID: "6dcfc392-e362-4d8a-8f26-f63dbecb07b5"). InnerVolumeSpecName "kube-api-access-w49s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:41.200680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.200646 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w49s4\" (UniqueName: \"kubernetes.io/projected/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-kube-api-access-w49s4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:41.200680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.200676 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dcfc392-e362-4d8a-8f26-f63dbecb07b5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:41.304124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.304083 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:41.304306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.304153 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:41.850497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.850436 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" Nov 03 17:27:41.850714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.850473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g" event={"ID":"6dcfc392-e362-4d8a-8f26-f63dbecb07b5","Type":"ContainerDied","Data":"489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368"} Nov 03 17:27:41.850714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:41.850600 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489cb102e211a65f2b356635972057dde2df646a0b84cda303ad0336c185d368" Nov 03 17:27:42.307572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:42.305970 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" start-of-body= Nov 03 17:27:42.307572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:42.306027 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.125:15021/healthz/ready\": dial tcp 10.134.1.125:15021: connect: connection refused" Nov 03 17:27:43.311133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.311048 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:27:43.312059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.312012 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" containerID="cri-o://941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101" gracePeriod=30 Nov 03 17:27:43.342181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.342145 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:27:43.342683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.342650 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" containerID="cri-o://4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f" gracePeriod=30 Nov 03 17:27:43.738938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.738901 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:43.739424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.739399 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:43.740388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:43.740364 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:27:47.469044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:47.469005 6001 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:27:47.469789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:47.469067 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:27:48.655753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.655718 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:27:48.704663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.704630 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:27:48.758084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758029 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.758401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758376 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.758682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758469 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.758682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758498 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.758682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758556 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.758682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758616 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.758682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758656 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758732 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758797 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758829 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.758967 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759003 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759041 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759120 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759178 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.759930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759273 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759351 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\") pod \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\" (UID: \"a70deee7-57c3-46ff-bebc-0ba1bec16c11\") " Nov 03 17:27:48.759930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.759380 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\") pod \"edc5fad7-2039-4555-b8c0-eba537123b4f\" (UID: \"edc5fad7-2039-4555-b8c0-eba537123b4f\") " Nov 03 17:27:48.761408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.760787 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.761408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.761195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:48.761408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.761237 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.761408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.761388 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:48.762029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.762001 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.762218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.762195 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.764075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.764047 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.765229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.761547 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.765491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.765429 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.765594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.765511 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.765771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.765743 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.767003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.766973 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.768001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.767975 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.768483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.768455 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.768483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.768466 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "kube-api-access-mn86s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.768655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.768483 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "kube-api-access-ctg6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.769420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.769392 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo" pod "edc5fad7-2039-4555-b8c0-eba537123b4f" (UID: "edc5fad7-2039-4555-b8c0-eba537123b4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:48.770013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.769992 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo" pod "a70deee7-57c3-46ff-bebc-0ba1bec16c11" (UID: "a70deee7-57c3-46ff-bebc-0ba1bec16c11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:48.862596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862507 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862541 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862556 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862573 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctg6b\" (UniqueName: \"kubernetes.io/projected/a70deee7-57c3-46ff-bebc-0ba1bec16c11-kube-api-access-ctg6b\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862595 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862608 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn86s\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-kube-api-access-mn86s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862617 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862625 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862633 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862641 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862649 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862657 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862666 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862673 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edc5fad7-2039-4555-b8c0-eba537123b4f-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862681 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862689 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edc5fad7-2039-4555-b8c0-eba537123b4f-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862697 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a70deee7-57c3-46ff-bebc-0ba1bec16c11-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.862892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.862705 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a70deee7-57c3-46ff-bebc-0ba1bec16c11-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.974286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.974247 6001 generic.go:356] "Generic (PLEG): container finished" podID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerID="941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101" exitCode=0 Nov 03 17:27:48.974492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.974308 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" Nov 03 17:27:48.974492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.974331 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerDied","Data":"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101"} Nov 03 17:27:48.974492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.974402 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg" event={"ID":"a70deee7-57c3-46ff-bebc-0ba1bec16c11","Type":"ContainerDied","Data":"cb09c518e39319d4eb48c6f27c97e94e4a4e78ab49e51f042d3d5cb64172d094"} Nov 03 17:27:48.974492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.974432 6001 scope.go:122] "RemoveContainer" containerID="941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101" Nov 03 17:27:48.976552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.976526 6001 generic.go:356] "Generic (PLEG): container finished" podID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerID="4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f" exitCode=0 Nov 03 17:27:48.976730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.976598 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" Nov 03 17:27:48.976730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.976648 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" event={"ID":"edc5fad7-2039-4555-b8c0-eba537123b4f","Type":"ContainerDied","Data":"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f"} Nov 03 17:27:48.976730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.976693 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl" event={"ID":"edc5fad7-2039-4555-b8c0-eba537123b4f","Type":"ContainerDied","Data":"e418885ebe3771adb34e885117fbaf5415fe2b67887f0bc56bb68f24cfe1db62"} Nov 03 17:27:48.986772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.986753 6001 scope.go:122] "RemoveContainer" containerID="fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e" Nov 03 17:27:48.998327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.998304 6001 scope.go:122] "RemoveContainer" containerID="941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101" Nov 03 17:27:48.998696 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:48.998675 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101\": container with ID starting with 941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101 not found: ID does not exist" containerID="941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101" Nov 03 17:27:48.998760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.998705 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101"} err="failed to get container status \"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101\": rpc error: code = NotFound desc = could not find container \"941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101\": container with ID starting with 941269cdffbf5b71ca421a48f3aca83841a7ad1a1a1b9e6ca7d74b02ee522101 not found: ID does not exist" Nov 03 17:27:48.998760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.998724 6001 scope.go:122] "RemoveContainer" containerID="fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e" Nov 03 17:27:48.998967 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:48.998949 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e\": container with ID starting with fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e not found: ID does not exist" containerID="fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e" Nov 03 17:27:48.999018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.998976 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e"} err="failed to get container status \"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e\": rpc error: code = NotFound desc = could not find container \"fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e\": container with ID starting with fcc2075d457d2fa156c561134490afaa2944ed76c4c41f697d8090c35d40559e not found: ID does not exist" Nov 03 17:27:48.999018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:48.998991 6001 scope.go:122] "RemoveContainer" containerID="4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f" Nov 03 17:27:49.009932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.009917 6001 scope.go:122] "RemoveContainer" containerID="4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f" Nov 03 17:27:49.010218 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:49.010195 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f\": container with ID starting with 4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f not found: ID does not exist" containerID="4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f" Nov 03 17:27:49.010313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.010222 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f"} err="failed to get container status \"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f\": rpc error: code = NotFound desc = could not find container \"4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f\": container with ID starting with 4f3ede60646aa389f3730ee82001a19741141408597b7bae0f4bd1e79ce5ec4f not found: ID does not exist" Nov 03 17:27:49.031401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.031363 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:27:49.043419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.043386 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-j54tl"] Nov 03 17:27:49.080301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.080263 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:27:49.105710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:49.105674 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-mvkrg"] Nov 03 17:27:50.798028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:50.797989 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" path="/var/lib/kubelet/pods/a70deee7-57c3-46ff-bebc-0ba1bec16c11/volumes" Nov 03 17:27:50.798976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:50.798951 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" path="/var/lib/kubelet/pods/edc5fad7-2039-4555-b8c0-eba537123b4f/volumes" Nov 03 17:27:51.024296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.024264 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:27:51.025044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025029 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6dcfc392-e362-4d8a-8f26-f63dbecb07b5" containerName="revoke-keys" Nov 03 17:27:51.025102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025046 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcfc392-e362-4d8a-8f26-f63dbecb07b5" containerName="revoke-keys" Nov 03 17:27:51.025102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025065 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.025102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025071 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.025102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025080 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.025102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025085 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025133 6001 cpu_manager.go:414] "containerMap: removing container" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025137 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025243 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc5fad7-2039-4555-b8c0-eba537123b4f" containerName="istio-proxy" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025252 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcfc392-e362-4d8a-8f26-f63dbecb07b5" containerName="revoke-keys" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025259 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.025268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.025265 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70deee7-57c3-46ff-bebc-0ba1bec16c11" containerName="istio-proxy" Nov 03 17:27:51.030536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.030515 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.047485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.047426 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06fa9cd7\"" Nov 03 17:27:51.051514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.051459 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:27:51.059111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.059087 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06fa9cd7-dockercfg-29rcm\"" Nov 03 17:27:51.135095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.135056 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbgw6\" (UniqueName: \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.165468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.163656 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.195409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.192559 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:27:51.198878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.198822 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.201040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.200933 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:27:51.279478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.276125 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbgw6\" (UniqueName: \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.279478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.276226 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.279478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.276289 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cs6\" (UniqueName: \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.279478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.276634 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.279478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.278014 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.306661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.306565 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbgw6\" (UniqueName: \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\") pod \"payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.345130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.345083 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:51.378109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.378071 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cs6\" (UniqueName: \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.378286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.378173 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.378911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.378886 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.404722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.403668 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cs6\" (UniqueName: \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\") pod \"payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.584299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.584207 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:51.660353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.660323 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:27:51.897557 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:27:51.897525 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3d69c2_fcb6_412d_9676_37cd88dd93de.slice/crio-93f8dabef487d0c77cb61f5df65d2e0e39319040f86e1f9d2b8f921d5aea3380 WatchSource:0}: Error finding container 93f8dabef487d0c77cb61f5df65d2e0e39319040f86e1f9d2b8f921d5aea3380: Status 404 returned error can't find the container with id 93f8dabef487d0c77cb61f5df65d2e0e39319040f86e1f9d2b8f921d5aea3380 Nov 03 17:27:51.898172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:51.898145 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:27:52.007733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.007699 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" event={"ID":"e0e311c2-651e-47af-9fbd-4bcec31aad87","Type":"ContainerStarted","Data":"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d"} Nov 03 17:27:52.007879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.007768 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" event={"ID":"e0e311c2-651e-47af-9fbd-4bcec31aad87","Type":"ContainerStarted","Data":"d5eb042f8a83a55f1f7b46f176161db5a2e734490436fdc2c01379cda01feeed"} Nov 03 17:27:52.007879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.007800 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:27:52.009581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.009551 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" event={"ID":"bd3d69c2-fcb6-412d-9676-37cd88dd93de","Type":"ContainerStarted","Data":"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad"} Nov 03 17:27:52.009676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.009606 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:27:52.009676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.009625 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" event={"ID":"bd3d69c2-fcb6-412d-9676-37cd88dd93de","Type":"ContainerStarted","Data":"93f8dabef487d0c77cb61f5df65d2e0e39319040f86e1f9d2b8f921d5aea3380"} Nov 03 17:27:52.054623 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.054562 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" podStartSLOduration=2.054547561 podStartE2EDuration="2.054547561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:52.050863955 +0000 UTC m=+4367.384077525" watchObservedRunningTime="2026-11-03 17:27:52.054547561 +0000 UTC m=+4367.387761129" Nov 03 17:27:52.096077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:27:52.096015 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" podStartSLOduration=1.095998954 podStartE2EDuration="1.095998954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:52.095769089 +0000 UTC m=+4367.428982660" watchObservedRunningTime="2026-11-03 17:27:52.095998954 +0000 UTC m=+4367.429212525" Nov 03 17:27:52.259232 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:27:52.259192 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:28:03.021078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:03.021042 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:28:03.021645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:03.021123 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:28:16.968646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:16.968598 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:28:16.969118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:16.968673 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:28:22.253898 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:28:22.253858 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:28:33.036825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:33.036779 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw"] Nov 03 17:28:33.047227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:33.047192 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt"] Nov 03 17:28:33.051573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:33.051539 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf8e168-s9nbw"] Nov 03 17:28:33.055590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:33.055554 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe63895-lccmt"] Nov 03 17:28:34.120612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.120573 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:28:34.126394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.126362 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.147061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.147024 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:28:34.192810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.192756 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193083 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6488\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193257 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193414 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193601 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193779 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.193979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.193961 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.194057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.194041 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.194116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.194105 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.196796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.196770 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.213803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.213774 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:28:34.218891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.218858 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:28:34.295956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.295906 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.295956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.295962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296069 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296111 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296305 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpv5p\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296357 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296522 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296560 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296588 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6488\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296618 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296678 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296727 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296775 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.296859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296811 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.296859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296856 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.297064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296910 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.297064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296920 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.297064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.296997 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.297196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.297072 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.297196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.297072 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.297350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.297328 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.298772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.298746 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.299291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.299270 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.320281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.320242 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.320439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.320375 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6488\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-t64tc\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.398363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398259 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398389 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398490 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpv5p\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398541 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398626 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398668 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398717 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.398786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398769 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398784 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398802 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.398995 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.399221 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.399273 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.399621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.399603 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.401178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.401155 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.401270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.401238 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.422121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.422088 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpv5p\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.422242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.422161 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-jmwhl\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.459294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.459256 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:34.520272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.520160 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:34.793717 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:28:34.793687 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4becb0_ff63_43f0_a20c_b4bf9700d700.slice/crio-dc2360099bc7c3e56416c5d3260bd24683a24a3ba97474156642ca5c86786664 WatchSource:0}: Error finding container dc2360099bc7c3e56416c5d3260bd24683a24a3ba97474156642ca5c86786664: Status 404 returned error can't find the container with id dc2360099bc7c3e56416c5d3260bd24683a24a3ba97474156642ca5c86786664 Nov 03 17:28:34.799588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.796464 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8" path="/var/lib/kubelet/pods/a2dcf4f2-4cc0-4442-a52c-d94e17b8dad8/volumes" Nov 03 17:28:34.799588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.797178 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebeaac82-b8c9-4639-9a59-1da50551fbc4" path="/var/lib/kubelet/pods/ebeaac82-b8c9-4639-9a59-1da50551fbc4/volumes" Nov 03 17:28:34.799588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.797655 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:28:34.802051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.802018 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:34.802168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.802112 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:34.802168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.802142 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:34.859881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.859845 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:28:34.862719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.862685 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:34.862839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.862759 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:34.862839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:34.862786 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:28:35.410487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.408955 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" event={"ID":"f988447e-72c6-4ea8-9b10-22d934337c34","Type":"ContainerStarted","Data":"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29"} Nov 03 17:28:35.410487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.409028 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" event={"ID":"f988447e-72c6-4ea8-9b10-22d934337c34","Type":"ContainerStarted","Data":"e3b7f52c064a9ecf6578c4250af67e198ff42d92ae19a9dda06347b02a2bbed5"} Nov 03 17:28:35.411123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.411088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" event={"ID":"ea4becb0-ff63-43f0-a20c-b4bf9700d700","Type":"ContainerStarted","Data":"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd"} Nov 03 17:28:35.411201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.411163 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" event={"ID":"ea4becb0-ff63-43f0-a20c-b4bf9700d700","Type":"ContainerStarted","Data":"dc2360099bc7c3e56416c5d3260bd24683a24a3ba97474156642ca5c86786664"} Nov 03 17:28:35.459704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.459664 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:35.462124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.462089 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:35.462333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.462152 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:35.464522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.464408 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podStartSLOduration=1.464393667 podStartE2EDuration="1.464393667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:35.458739216 +0000 UTC m=+4410.791952785" watchObservedRunningTime="2026-11-03 17:28:35.464393667 +0000 UTC m=+4410.797607273" Nov 03 17:28:35.509771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.509713 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podStartSLOduration=1.509695179 podStartE2EDuration="1.509695179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:35.505550357 +0000 UTC m=+4410.838763927" watchObservedRunningTime="2026-11-03 17:28:35.509695179 +0000 UTC m=+4410.842908747" Nov 03 17:28:35.523315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.521040 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:35.523315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.523254 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:35.523637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:35.523318 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:36.462483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:36.461918 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:36.462483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:36.462005 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:36.520716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:36.520470 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:36.520716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:36.520520 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:37.463472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:37.461960 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:37.463472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:37.462050 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:37.520771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:37.520731 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:37.521019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:37.520807 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:38.470469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:38.470151 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:38.470469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:38.470214 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:38.520665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:38.520628 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:38.520890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:38.520697 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:39.461468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:39.460892 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:39.461468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:39.460981 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:39.520678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:39.520644 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:39.521105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:39.520699 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:40.463470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:40.459914 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:40.463470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:40.459984 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:40.520866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:40.520834 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:40.521393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:40.520891 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:41.460710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:41.460666 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:41.460914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:41.460735 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:41.521172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:41.521127 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:41.521667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:41.521192 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:42.460512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:42.460472 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-t64tc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" start-of-body= Nov 03 17:28:42.460736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:42.460541 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.129:15021/healthz/ready\": dial tcp 10.134.1.129:15021: connect: connection refused" Nov 03 17:28:42.524433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:42.524392 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" start-of-body= Nov 03 17:28:42.525165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:42.525119 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.130:15021/healthz/ready\": dial tcp 10.134.1.130:15021: connect: connection refused" Nov 03 17:28:43.877571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.877538 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:43.878146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.878068 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:43.879175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.879150 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:28:43.935731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.935684 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:43.936172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.936148 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:43.937181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:43.937159 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:28:44.048857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.045594 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8"] Nov 03 17:28:44.055048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.054981 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576607e-m82q8"] Nov 03 17:28:44.078496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.062414 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw"] Nov 03 17:28:44.078496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.071846 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1539e8a-8k8cw"] Nov 03 17:28:44.790842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.790812 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7081157f-d26e-4812-8db2-94875fb35a24" path="/var/lib/kubelet/pods/7081157f-d26e-4812-8db2-94875fb35a24/volumes" Nov 03 17:28:44.791261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:28:44.791247 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cacc4860-3437-4210-beaa-592d39869526" path="/var/lib/kubelet/pods/cacc4860-3437-4210-beaa-592d39869526/volumes" Nov 03 17:28:52.252898 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:28:52.252809 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:29:18.535561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.535525 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7"] Nov 03 17:29:18.540272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.540248 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.560162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.560128 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7"] Nov 03 17:29:18.568310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.568263 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:29:18.568310 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.568289 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:29:18.568603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.568357 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:29:18.568603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.568362 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:29:18.644962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.644926 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrvc\" (UniqueName: \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.645190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.645072 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.645190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.645113 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.746802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.746762 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrvc\" (UniqueName: \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.746992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.746864 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.746992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.746891 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.747647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.747618 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.749465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.749423 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.770706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.770673 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrvc\" (UniqueName: \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\") pod \"maas-api-revoke-sub-f9dca08c-s5qv7\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:18.856236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:18.856159 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:19.172358 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:29:19.172323 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacd2345_cd1c_4d5c_a3f7_b4e55320efcb.slice/crio-d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48 WatchSource:0}: Error finding container d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48: Status 404 returned error can't find the container with id d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48 Nov 03 17:29:19.175648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:19.175620 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7"] Nov 03 17:29:19.934938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:19.934836 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" event={"ID":"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb","Type":"ContainerStarted","Data":"89a922f5745ceaa0e9d1abbec147d04a0f8cf0efa0ab8331e1323d924379ae83"} Nov 03 17:29:19.934938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:19.934929 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" event={"ID":"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb","Type":"ContainerStarted","Data":"d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48"} Nov 03 17:29:19.989802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:19.989738 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" podStartSLOduration=1.989724617 podStartE2EDuration="1.989724617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:29:19.988095627 +0000 UTC m=+4455.321309197" watchObservedRunningTime="2026-11-03 17:29:19.989724617 +0000 UTC m=+4455.322938183" Nov 03 17:29:20.945208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:20.945169 6001 generic.go:356] "Generic (PLEG): container finished" podID="dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" containerID="89a922f5745ceaa0e9d1abbec147d04a0f8cf0efa0ab8331e1323d924379ae83" exitCode=0 Nov 03 17:29:20.945754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:20.945251 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" event={"ID":"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb","Type":"ContainerDied","Data":"89a922f5745ceaa0e9d1abbec147d04a0f8cf0efa0ab8331e1323d924379ae83"} Nov 03 17:29:22.134755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.134727 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:22.208939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.208838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\") pod \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " Nov 03 17:29:22.209204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.209182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\" (UniqueName: \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\") pod \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " Nov 03 17:29:22.209282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.209240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\") pod \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\" (UID: \"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb\") " Nov 03 17:29:22.209282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.209257 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca" pod "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" (UID: "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:22.209820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.209797 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:22.211607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.211583 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca" pod "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" (UID: "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:29:22.212410 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.212381 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc" pod "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" (UID: "dacd2345-cd1c-4d5c-a3f7-b4e55320efcb"). InnerVolumeSpecName "kube-api-access-lzrvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:22.254323 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:29:22.254286 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:29:22.311730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.311682 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzrvc\" (UniqueName: \"kubernetes.io/projected/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-kube-api-access-lzrvc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:22.311730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.311720 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:22.964513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.964486 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" Nov 03 17:29:22.964693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.964556 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7" event={"ID":"dacd2345-cd1c-4d5c-a3f7-b4e55320efcb","Type":"ContainerDied","Data":"d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48"} Nov 03 17:29:22.964693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:22.964606 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8371b54af5ea97c76d7fbc63b9fe5e2929e72a680f08562fb95cd3bf4289a48" Nov 03 17:29:35.236276 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:35.236236 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:29:35.236770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:35.236561 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" containerID="cri-o://0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d" gracePeriod=30 Nov 03 17:29:40.529554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.529517 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:29:40.640803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.640759 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.640819 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.640902 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641086 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641140 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641182 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641269 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641319 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.641378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.641365 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\") pod \"3756ab63-6407-41c4-8b8b-8596cb24e43b\" (UID: \"3756ab63-6407-41c4-8b8b-8596cb24e43b\") " Nov 03 17:29:40.642336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.642145 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.643041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.642590 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.643041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.642677 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:40.643041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.642693 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.643041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.642939 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.644237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.644210 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "kube-api-access-nbmt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:40.645781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.645756 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:29:40.645947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.645911 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:29:40.646505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.646438 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token" pod "3756ab63-6407-41c4-8b8b-8596cb24e43b" (UID: "3756ab63-6407-41c4-8b8b-8596cb24e43b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.743657 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3756ab63-6407-41c4-8b8b-8596cb24e43b-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744243 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744262 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744277 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744301 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744316 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744329 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744344 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmt7\" (UniqueName: \"kubernetes.io/projected/3756ab63-6407-41c4-8b8b-8596cb24e43b-kube-api-access-nbmt7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:40.744436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:40.744366 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3756ab63-6407-41c4-8b8b-8596cb24e43b-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:29:41.170934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.170840 6001 generic.go:356] "Generic (PLEG): container finished" podID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerID="0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d" exitCode=0 Nov 03 17:29:41.170934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.170920 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" Nov 03 17:29:41.171220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.170939 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" event={"ID":"3756ab63-6407-41c4-8b8b-8596cb24e43b","Type":"ContainerDied","Data":"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d"} Nov 03 17:29:41.171220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.171008 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln" event={"ID":"3756ab63-6407-41c4-8b8b-8596cb24e43b","Type":"ContainerDied","Data":"0f70b4e13491821475b0c63a4134856491029288a369ed0bba4e499680b0745c"} Nov 03 17:29:41.171220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.171038 6001 scope.go:122] "RemoveContainer" containerID="0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d" Nov 03 17:29:41.189000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.188972 6001 scope.go:122] "RemoveContainer" containerID="0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d" Nov 03 17:29:41.189343 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:29:41.189321 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d\": container with ID starting with 0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d not found: ID does not exist" containerID="0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d" Nov 03 17:29:41.189400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.189356 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d"} err="failed to get container status \"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d\": rpc error: code = NotFound desc = could not find container \"0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d\": container with ID starting with 0299f28bf4a05b39749c9148a103a12f2bb1353b390c3665429d5da8c66ae98d not found: ID does not exist" Nov 03 17:29:41.227901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.227860 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:29:41.240495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:41.240455 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bwtln"] Nov 03 17:29:42.791132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:42.791095 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" path="/var/lib/kubelet/pods/3756ab63-6407-41c4-8b8b-8596cb24e43b/volumes" Nov 03 17:29:42.863098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:42.863056 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:29:42.896557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:42.896517 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:29:47.864194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:47.864131 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" containerID="cri-o://417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d" gracePeriod=25 Nov 03 17:29:47.897153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:47.897115 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" containerID="cri-o://57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4" gracePeriod=25 Nov 03 17:29:50.042005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:50.041963 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m"] Nov 03 17:29:50.054245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:50.054201 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-082500be-mr29m"] Nov 03 17:29:50.791356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:50.791318 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366af900-f825-402b-b013-b93794ec3c68" path="/var/lib/kubelet/pods/366af900-f825-402b-b013-b93794ec3c68/volumes" Nov 03 17:29:51.029476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:51.029429 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7"] Nov 03 17:29:51.041643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:51.041567 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48545c07-nwmh7"] Nov 03 17:29:52.253144 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:29:52.253107 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:29:52.790647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:52.790609 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6" path="/var/lib/kubelet/pods/a71f63d7-68a1-4cc8-a6ca-3a3bbb0964c6/volumes" Nov 03 17:29:53.735464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:53.734716 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:29:53.737842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:29:53.736538 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:00.163923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.163887 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:00.164996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.164970 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" Nov 03 17:30:00.164996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.164994 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" Nov 03 17:30:00.165207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.165018 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" containerName="revoke-keys" Nov 03 17:30:00.165207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.165025 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" containerName="revoke-keys" Nov 03 17:30:00.165207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.165200 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3756ab63-6407-41c4-8b8b-8596cb24e43b" containerName="istio-proxy" Nov 03 17:30:00.165368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.165218 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" containerName="revoke-keys" Nov 03 17:30:00.169989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.169961 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:00.170125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.170100 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.175511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.175491 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:00.175620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.175601 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.181346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.181328 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:00.181453 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.181427 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.186922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.186899 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:00.187055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.187023 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.190295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.190046 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:30:00.190572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.190549 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:30:00.193554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.193530 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:30:00.193873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.193836 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.201174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.201146 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:00.213290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.212733 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:00.213290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.212784 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:00.213290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.212883 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.246407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246371 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.246643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246535 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.246643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246637 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.246787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246664 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgdv\" (UniqueName: \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.246787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246733 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.246787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246782 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.246966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246838 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztbg8\" (UniqueName: \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.246966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246894 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcq5\" (UniqueName: \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.246966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246918 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqvc\" (UniqueName: \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.247140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.246975 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.247140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.247004 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ctsq\" (UniqueName: \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.247140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.247028 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pqp\" (UniqueName: \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.251869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.249720 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:00.256854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.256151 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:00.265616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.263631 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:00.265616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.265009 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.278684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.275466 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:00.282154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.279636 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.291740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.287957 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:00.293903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.293876 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.298198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.298016 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:00.298198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.298128 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.306019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.304037 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:00.310639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.310611 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:00.310791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.310770 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.315752 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.315727 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:00.320762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.320731 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:00.331216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.331187 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349107 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcq5\" (UniqueName: \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349157 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqvc\" (UniqueName: \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349208 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349240 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ctsq\" (UniqueName: \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349265 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pqp\" (UniqueName: \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349339 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxt8\" (UniqueName: \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.349436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.349376 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.350187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350155 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqzp\" (UniqueName: \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.350296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350205 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwpx\" (UniqueName: \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.350296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350253 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.350426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.350426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350371 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.350426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350406 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.350683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350460 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.350683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350525 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.350683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350579 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.350683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350611 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgdv\" (UniqueName: \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.350683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350649 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z5w8\" (UniqueName: \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.350987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350687 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.350987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350726 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.350987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.350987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.350168 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.351525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.351456 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.351625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.351548 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.352217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.352192 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.352459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.352423 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.352851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.352826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztbg8\" (UniqueName: \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.352942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.352855 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.364016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.363994 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:00.366525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.364235 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.374086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.374060 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:00.374304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.374177 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.374404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.374311 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:00.374404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.374355 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:00.381640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.381618 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:00.385437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.381720 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.390071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.390047 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:00.390189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.390106 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:00.390189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.390052 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.394376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.394351 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqvc\" (UniqueName: \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\") pod \"maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.394489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.394376 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ctsq\" (UniqueName: \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\") pod \"maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.394489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.394353 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pqp\" (UniqueName: \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\") pod \"maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.394665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.394638 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:00.394771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.394751 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.395563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.395533 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgdv\" (UniqueName: \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\") pod \"maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.395792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.395771 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztbg8\" (UniqueName: \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\") pod \"maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.396086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.396068 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcq5\" (UniqueName: \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\") pod \"maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.403343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.403320 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:00.407283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.407259 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:00.410750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.410727 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:00.417179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.417125 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:00.417257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.417233 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.419726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.419704 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:00.423670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.423649 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:00.442758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.442678 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.443205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.443188 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:00.453957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.453927 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gwk\" (UniqueName: \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.454220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.453992 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774nl\" (UniqueName: \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.454220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454034 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqzp\" (UniqueName: \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.454220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454056 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwpx\" (UniqueName: \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.454220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454090 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkt7w\" (UniqueName: \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.454485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454237 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.454485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454274 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.454485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454300 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.454485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454328 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.454485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454397 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454493 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454537 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454578 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454631 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454675 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6dq\" (UniqueName: \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454721 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z5w8\" (UniqueName: \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.454775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454772 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.455163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454799 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.455163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454968 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.455163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.454975 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.455163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455048 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvztv\" (UniqueName: \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.455163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455045 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.455435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455208 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc26m\" (UniqueName: \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.455435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455249 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:00.455435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455299 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxt8\" (UniqueName: \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.455683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455660 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.455779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.455725 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.456075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.456050 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.459393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.459369 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:00.488901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.488874 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwpx\" (UniqueName: \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.490018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.489998 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z5w8\" (UniqueName: \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\") pod \"maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.490689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.490668 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxt8\" (UniqueName: \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.490797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.490785 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:00.500102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.500080 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.500347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.500333 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqzp\" (UniqueName: \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.529327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.529291 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:00.546939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.546907 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.556877 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gwk\" (UniqueName: \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.556948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774nl\" (UniqueName: \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557007 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkt7w\" (UniqueName: \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557067 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557100 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557205 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557240 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.557291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557279 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557345 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6dq\" (UniqueName: \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557419 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557561 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557592 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvztv\" (UniqueName: \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557644 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnfhs\" (UniqueName: \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.557798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.557695 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc26m\" (UniqueName: \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.559563 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.559580 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.559563 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.560114 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.560277 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.560674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.560426 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.563260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.563088 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:00.578760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.578729 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:00.587932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.586664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvztv\" (UniqueName: \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.589506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.588837 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774nl\" (UniqueName: \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.589506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.589194 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkt7w\" (UniqueName: \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.589506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.589461 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc26m\" (UniqueName: \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\") pod \"maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.591105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.591060 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gwk\" (UniqueName: \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.595107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.594860 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:00.606378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.606300 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6dq\" (UniqueName: \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.614166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.613923 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:00.628422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.628041 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:00.648652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.646243 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:00.660620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.660032 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.660620 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.660268 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnfhs\" (UniqueName: \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.662379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.662353 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:00.672077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.671666 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.672077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.671904 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:00.702264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.702232 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnfhs\" (UniqueName: \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.748973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.748942 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:00.755664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.755223 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:00.760301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.760081 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:00.770566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.764001 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:00.772574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.772505 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:00.823290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.780800 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:00.823290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.803012 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:00.907133 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:00.907091 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02a0e58_8cb5_4057_ac74_4b52d64f3176.slice/crio-b608ef700551f6bcde0bd287836f05dee43cfdb164bf671515b7a2340b0fa667 WatchSource:0}: Error finding container b608ef700551f6bcde0bd287836f05dee43cfdb164bf671515b7a2340b0fa667: Status 404 returned error can't find the container with id b608ef700551f6bcde0bd287836f05dee43cfdb164bf671515b7a2340b0fa667 Nov 03 17:30:00.939329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:00.939293 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:01.060598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.054523 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8"] Nov 03 17:30:01.075038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.074952 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d4bb8b2-vb5k8"] Nov 03 17:30:01.277208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.277180 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:01.296199 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.296165 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02617808_3d0b_4cbb_ac17_ec67fbac6ae1.slice/crio-85100d9095f3e830c271e83cc672c3244716936678082de6daa74414c44ff092 WatchSource:0}: Error finding container 85100d9095f3e830c271e83cc672c3244716936678082de6daa74414c44ff092: Status 404 returned error can't find the container with id 85100d9095f3e830c271e83cc672c3244716936678082de6daa74414c44ff092 Nov 03 17:30:01.308342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.308308 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:01.363583 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.363548 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cf3eae_23d4_4b97_bcf0_e6d6ebe4f63c.slice/crio-13b879b90ebec05d0b55be35093ecef959a53ddde831a089db297190f0e7ad64 WatchSource:0}: Error finding container 13b879b90ebec05d0b55be35093ecef959a53ddde831a089db297190f0e7ad64: Status 404 returned error can't find the container with id 13b879b90ebec05d0b55be35093ecef959a53ddde831a089db297190f0e7ad64 Nov 03 17:30:01.365428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.365407 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:01.397278 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.397249 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8565be49_1874_41cd_b659_aa5a09a24c74.slice/crio-1e9020a8a082102d76ebc0b284e0d3541408997dcfca44896a370c4ed939e6f2 WatchSource:0}: Error finding container 1e9020a8a082102d76ebc0b284e0d3541408997dcfca44896a370c4ed939e6f2: Status 404 returned error can't find the container with id 1e9020a8a082102d76ebc0b284e0d3541408997dcfca44896a370c4ed939e6f2 Nov 03 17:30:01.398959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.398933 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerStarted","Data":"b608ef700551f6bcde0bd287836f05dee43cfdb164bf671515b7a2340b0fa667"} Nov 03 17:30:01.399073 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.399000 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:01.400686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.400662 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerStarted","Data":"4361c46c2489947154ed33d0ae104aef80886278b254d468e3326c44cae6fe80"} Nov 03 17:30:01.401136 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.401106 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47398737_8736_4f4b_b8d4_9ba4d716ea06.slice/crio-401fe9629c631484afde356919414a08ef5be4d1e8e621d2fce6b4b3c075c2fa WatchSource:0}: Error finding container 401fe9629c631484afde356919414a08ef5be4d1e8e621d2fce6b4b3c075c2fa: Status 404 returned error can't find the container with id 401fe9629c631484afde356919414a08ef5be4d1e8e621d2fce6b4b3c075c2fa Nov 03 17:30:01.405061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.404072 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerStarted","Data":"85100d9095f3e830c271e83cc672c3244716936678082de6daa74414c44ff092"} Nov 03 17:30:01.406221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.405895 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:01.408229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.408204 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerStarted","Data":"13b879b90ebec05d0b55be35093ecef959a53ddde831a089db297190f0e7ad64"} Nov 03 17:30:01.412017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.411993 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:01.415576 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.415547 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426d8922_d11b_4983_8b2d_52467d60dade.slice/crio-c63bcd26fa9411e1009b6432a1d27ff90e3a79c1146a55eb7b610968d7a3cfb1 WatchSource:0}: Error finding container c63bcd26fa9411e1009b6432a1d27ff90e3a79c1146a55eb7b610968d7a3cfb1: Status 404 returned error can't find the container with id c63bcd26fa9411e1009b6432a1d27ff90e3a79c1146a55eb7b610968d7a3cfb1 Nov 03 17:30:01.517439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.517411 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:01.526222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.526194 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:01.555116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.555084 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:01.560004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.559975 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:01.564474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.564439 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:01.568964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.568940 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:01.573925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.573899 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:01.626992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.626965 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:01.630719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.630697 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:01.645874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.645845 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:01.649858 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.649833 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fac7a2_4eda_4e5c_be12_531425e2e807.slice/crio-30ca6428a83c0a69b7127d95ab94ea411e9eb6baf075e5b75ab147a09e768c6c WatchSource:0}: Error finding container 30ca6428a83c0a69b7127d95ab94ea411e9eb6baf075e5b75ab147a09e768c6c: Status 404 returned error can't find the container with id 30ca6428a83c0a69b7127d95ab94ea411e9eb6baf075e5b75ab147a09e768c6c Nov 03 17:30:01.651432 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.651409 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d0bf64_0564_4905_8d4d_b932579d9f69.slice/crio-727b43df676128f112d5cb5b15a4c159749d4ef0d7c537121f2a99208f52f098 WatchSource:0}: Error finding container 727b43df676128f112d5cb5b15a4c159749d4ef0d7c537121f2a99208f52f098: Status 404 returned error can't find the container with id 727b43df676128f112d5cb5b15a4c159749d4ef0d7c537121f2a99208f52f098 Nov 03 17:30:01.654621 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.653103 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff5ae3c_7c28_454a_8dfe_a1a6872cb1d9.slice/crio-8dc35b9b5261f507768042bb49399312ba641eabf75d6ccc8d1ad847796e91c8 WatchSource:0}: Error finding container 8dc35b9b5261f507768042bb49399312ba641eabf75d6ccc8d1ad847796e91c8: Status 404 returned error can't find the container with id 8dc35b9b5261f507768042bb49399312ba641eabf75d6ccc8d1ad847796e91c8 Nov 03 17:30:01.654910 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.654734 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1547f64_57e7_4641_bee9_c1449c90b12f.slice/crio-7a24cc17ddf39af88a8ec20d1332e4e88bf4f22cc7a6021aabc88b6e105f2068 WatchSource:0}: Error finding container 7a24cc17ddf39af88a8ec20d1332e4e88bf4f22cc7a6021aabc88b6e105f2068: Status 404 returned error can't find the container with id 7a24cc17ddf39af88a8ec20d1332e4e88bf4f22cc7a6021aabc88b6e105f2068 Nov 03 17:30:01.657533 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.657510 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e24410_4d9e_4116_93b5_aa72aac5672e.slice/crio-9ca62a4f2e51a8048176dbe55947ec4655542f70d5ace9384a90135626590f73 WatchSource:0}: Error finding container 9ca62a4f2e51a8048176dbe55947ec4655542f70d5ace9384a90135626590f73: Status 404 returned error can't find the container with id 9ca62a4f2e51a8048176dbe55947ec4655542f70d5ace9384a90135626590f73 Nov 03 17:30:01.658680 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.658655 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddbd6bf_5f2b_47a6_90c6_85daf77b6839.slice/crio-0bd3d464d34fab2e6b36a282bb411e1ec98ca7fac6976a7b1291cd327fe023e5 WatchSource:0}: Error finding container 0bd3d464d34fab2e6b36a282bb411e1ec98ca7fac6976a7b1291cd327fe023e5: Status 404 returned error can't find the container with id 0bd3d464d34fab2e6b36a282bb411e1ec98ca7fac6976a7b1291cd327fe023e5 Nov 03 17:30:01.660273 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.660251 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d7b1e1_09a5_4273_9cc0_2a278231ddf2.slice/crio-bb73f17266719d10306b33b916e530e15fbb07b3f26f2a2e814fc06f14cde697 WatchSource:0}: Error finding container bb73f17266719d10306b33b916e530e15fbb07b3f26f2a2e814fc06f14cde697: Status 404 returned error can't find the container with id bb73f17266719d10306b33b916e530e15fbb07b3f26f2a2e814fc06f14cde697 Nov 03 17:30:01.662023 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.662003 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c917b6_5e0b_4235_85a3_55acd85f2f6f.slice/crio-36a336b0472481fe2212680937b532cfdae1f8e6ee4ece01c02889c762fc0f0d WatchSource:0}: Error finding container 36a336b0472481fe2212680937b532cfdae1f8e6ee4ece01c02889c762fc0f0d: Status 404 returned error can't find the container with id 36a336b0472481fe2212680937b532cfdae1f8e6ee4ece01c02889c762fc0f0d Nov 03 17:30:01.693892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:01.693861 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:01.720643 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:30:01.720602 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa32e18_1a18_43ea_be00_01871c323b4c.slice/crio-fd7cd88db010bc4786fab47902ab7b38aeb6e13198cbd80b6e028fbc1d6a3b79 WatchSource:0}: Error finding container fd7cd88db010bc4786fab47902ab7b38aeb6e13198cbd80b6e028fbc1d6a3b79: Status 404 returned error can't find the container with id fd7cd88db010bc4786fab47902ab7b38aeb6e13198cbd80b6e028fbc1d6a3b79 Nov 03 17:30:02.418353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.418321 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerStarted","Data":"7a24cc17ddf39af88a8ec20d1332e4e88bf4f22cc7a6021aabc88b6e105f2068"} Nov 03 17:30:02.419799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.419762 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerStarted","Data":"727b43df676128f112d5cb5b15a4c159749d4ef0d7c537121f2a99208f52f098"} Nov 03 17:30:02.421097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.421072 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerStarted","Data":"9d31451edf4143c58ea331d427beeac395459aa9d5e507065698c28c7b7f1570"} Nov 03 17:30:02.422299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.422276 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerStarted","Data":"434bb0060558ac41efd9feff3a21a4b14c7aec9238cb2e47b2d365535a8881d9"} Nov 03 17:30:02.423665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.423640 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerStarted","Data":"30ca6428a83c0a69b7127d95ab94ea411e9eb6baf075e5b75ab147a09e768c6c"} Nov 03 17:30:02.424982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.424958 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerStarted","Data":"c63bcd26fa9411e1009b6432a1d27ff90e3a79c1146a55eb7b610968d7a3cfb1"} Nov 03 17:30:02.426357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.426330 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerStarted","Data":"8dc35b9b5261f507768042bb49399312ba641eabf75d6ccc8d1ad847796e91c8"} Nov 03 17:30:02.428030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.428004 6001 generic.go:356] "Generic (PLEG): container finished" podID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerID="3df41bdb6d293d3b9f1872a672a4c59e1d25776a7c17d3d5ef106adbce700b26" exitCode=60 Nov 03 17:30:02.428122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.428052 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerDied","Data":"3df41bdb6d293d3b9f1872a672a4c59e1d25776a7c17d3d5ef106adbce700b26"} Nov 03 17:30:02.428499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.428477 6001 scope.go:122] "RemoveContainer" containerID="3df41bdb6d293d3b9f1872a672a4c59e1d25776a7c17d3d5ef106adbce700b26" Nov 03 17:30:02.429909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.429887 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerStarted","Data":"5401d0c22d69ee333f8305ba0b2e84fa3fc3a358f91047bfcae94f9f10e04451"} Nov 03 17:30:02.431319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.431291 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerStarted","Data":"0bd3d464d34fab2e6b36a282bb411e1ec98ca7fac6976a7b1291cd327fe023e5"} Nov 03 17:30:02.432705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.432680 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerStarted","Data":"bb73f17266719d10306b33b916e530e15fbb07b3f26f2a2e814fc06f14cde697"} Nov 03 17:30:02.434022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.434002 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerStarted","Data":"36a336b0472481fe2212680937b532cfdae1f8e6ee4ece01c02889c762fc0f0d"} Nov 03 17:30:02.435319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.435289 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerStarted","Data":"9ca62a4f2e51a8048176dbe55947ec4655542f70d5ace9384a90135626590f73"} Nov 03 17:30:02.436880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.436830 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerStarted","Data":"1e9020a8a082102d76ebc0b284e0d3541408997dcfca44896a370c4ed939e6f2"} Nov 03 17:30:02.443784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.443695 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerStarted","Data":"75d670683578a3547d8556177a350dd02eaaf7f47634a94343204201f96b2676"} Nov 03 17:30:02.460261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.460210 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerStarted","Data":"548ab20db5cfc0258f1a95f598e4b9c0a0c0e67c3d2e99da51af71c1764847cf"} Nov 03 17:30:02.460751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.460300 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerStarted","Data":"401fe9629c631484afde356919414a08ef5be4d1e8e621d2fce6b4b3c075c2fa"} Nov 03 17:30:02.471355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.471300 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerStarted","Data":"fd7cd88db010bc4786fab47902ab7b38aeb6e13198cbd80b6e028fbc1d6a3b79"} Nov 03 17:30:02.530017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.529904 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" podStartSLOduration=2.529884038 podStartE2EDuration="2.529884038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.523554612 +0000 UTC m=+4497.856768212" watchObservedRunningTime="2026-11-03 17:30:02.529884038 +0000 UTC m=+4497.863097608" Nov 03 17:30:02.566362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.566307 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" podStartSLOduration=2.56629143 podStartE2EDuration="2.56629143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.565954794 +0000 UTC m=+4497.899168373" watchObservedRunningTime="2026-11-03 17:30:02.56629143 +0000 UTC m=+4497.899505000" Nov 03 17:30:02.610090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.610035 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" podStartSLOduration=2.6100230250000003 podStartE2EDuration="2.610023025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.606851873 +0000 UTC m=+4497.940065443" watchObservedRunningTime="2026-11-03 17:30:02.610023025 +0000 UTC m=+4497.943236604" Nov 03 17:30:02.793520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:02.793437 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2af11a6-30f1-41b4-857f-25967bc2785d" path="/var/lib/kubelet/pods/c2af11a6-30f1-41b4-857f-25967bc2785d/volumes" Nov 03 17:30:03.483051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.483008 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerStarted","Data":"3ed8c41bf9c7cb0136291898c47b456a6ff92c5f3f0ab29264124f98c0ec36c6"} Nov 03 17:30:03.484890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.484862 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerStarted","Data":"6a4a6c0a13725f96d6b80c381f106aad1b77d16ee149654c950946efb90bcefe"} Nov 03 17:30:03.486670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.486648 6001 generic.go:356] "Generic (PLEG): container finished" podID="426d8922-d11b-4983-8b2d-52467d60dade" containerID="0d0148c4e908e00b2ada9eb020b5b206ffb8b73c88ef3dba4c34ee6aa38ec41a" exitCode=60 Nov 03 17:30:03.486762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.486718 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerDied","Data":"0d0148c4e908e00b2ada9eb020b5b206ffb8b73c88ef3dba4c34ee6aa38ec41a"} Nov 03 17:30:03.486925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.486907 6001 scope.go:122] "RemoveContainer" containerID="0d0148c4e908e00b2ada9eb020b5b206ffb8b73c88ef3dba4c34ee6aa38ec41a" Nov 03 17:30:03.488438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.488411 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerStarted","Data":"a5c45aae9f8728c80496a1ecbdde2b5498692dbe3ddd2842ad9b8ef0b6597aac"} Nov 03 17:30:03.490045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.490023 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerStarted","Data":"9fa623ddd32e4691b6200272a96132ef3373f39f2595d40cf50a97d369079a96"} Nov 03 17:30:03.491696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.491672 6001 generic.go:356] "Generic (PLEG): container finished" podID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerID="5401d0c22d69ee333f8305ba0b2e84fa3fc3a358f91047bfcae94f9f10e04451" exitCode=60 Nov 03 17:30:03.491781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.491748 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerDied","Data":"5401d0c22d69ee333f8305ba0b2e84fa3fc3a358f91047bfcae94f9f10e04451"} Nov 03 17:30:03.491921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.491906 6001 scope.go:122] "RemoveContainer" containerID="5401d0c22d69ee333f8305ba0b2e84fa3fc3a358f91047bfcae94f9f10e04451" Nov 03 17:30:03.493368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.493345 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerStarted","Data":"156ebe1bad659e0df8c9d8d27ae4096c7da1ef316fc3537463759147ff572969"} Nov 03 17:30:03.494976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.494954 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerStarted","Data":"64beab449167725e49248ea05f6ff86437f2e23239e1725e6b8c32957df70d01"} Nov 03 17:30:03.496586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.496560 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerStarted","Data":"0e4b0ac443b2982e53532481d4a396a20c78b865017c1c97606f16aaef32aecd"} Nov 03 17:30:03.498102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.498082 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerStarted","Data":"5ec8956f3dacd9d338b0a0b0abc8952e320bf75c340aad903c14e1d02a43a147"} Nov 03 17:30:03.499728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.499708 6001 generic.go:356] "Generic (PLEG): container finished" podID="8565be49-1874-41cd-b659-aa5a09a24c74" containerID="466c3aeeaf58af5975ae90873e6657cef8b6c3b2d7183689d7845dece46a2848" exitCode=60 Nov 03 17:30:03.499801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.499767 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerDied","Data":"466c3aeeaf58af5975ae90873e6657cef8b6c3b2d7183689d7845dece46a2848"} Nov 03 17:30:03.499973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.499953 6001 scope.go:122] "RemoveContainer" containerID="466c3aeeaf58af5975ae90873e6657cef8b6c3b2d7183689d7845dece46a2848" Nov 03 17:30:03.501391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.501371 6001 generic.go:356] "Generic (PLEG): container finished" podID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerID="75d670683578a3547d8556177a350dd02eaaf7f47634a94343204201f96b2676" exitCode=60 Nov 03 17:30:03.501509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.501490 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerDied","Data":"75d670683578a3547d8556177a350dd02eaaf7f47634a94343204201f96b2676"} Nov 03 17:30:03.501631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.501618 6001 scope.go:122] "RemoveContainer" containerID="75d670683578a3547d8556177a350dd02eaaf7f47634a94343204201f96b2676" Nov 03 17:30:03.503039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.503022 6001 generic.go:356] "Generic (PLEG): container finished" podID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerID="548ab20db5cfc0258f1a95f598e4b9c0a0c0e67c3d2e99da51af71c1764847cf" exitCode=60 Nov 03 17:30:03.503117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.503098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerDied","Data":"548ab20db5cfc0258f1a95f598e4b9c0a0c0e67c3d2e99da51af71c1764847cf"} Nov 03 17:30:03.503260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.503249 6001 scope.go:122] "RemoveContainer" containerID="548ab20db5cfc0258f1a95f598e4b9c0a0c0e67c3d2e99da51af71c1764847cf" Nov 03 17:30:03.527363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.527304 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" podStartSLOduration=3.527289365 podStartE2EDuration="3.527289365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.525473474 +0000 UTC m=+4498.858687044" watchObservedRunningTime="2026-11-03 17:30:03.527289365 +0000 UTC m=+4498.860502934" Nov 03 17:30:03.612920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.612802 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" podStartSLOduration=3.612789338 podStartE2EDuration="3.612789338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.607957144 +0000 UTC m=+4498.941170714" watchObservedRunningTime="2026-11-03 17:30:03.612789338 +0000 UTC m=+4498.946003272" Nov 03 17:30:03.660761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.659895 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" podStartSLOduration=3.65987347 podStartE2EDuration="3.65987347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.651737129 +0000 UTC m=+4498.984950695" watchObservedRunningTime="2026-11-03 17:30:03.65987347 +0000 UTC m=+4498.993087042" Nov 03 17:30:03.697429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.697371 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" podStartSLOduration=3.697351835 podStartE2EDuration="3.697351835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.696847325 +0000 UTC m=+4499.030060895" watchObservedRunningTime="2026-11-03 17:30:03.697351835 +0000 UTC m=+4499.030565409" Nov 03 17:30:03.714673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.714635 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:03.715126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.715101 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:03.743818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.743752 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" podStartSLOduration=3.743733281 podStartE2EDuration="3.743733281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.73930394 +0000 UTC m=+4499.072517510" watchObservedRunningTime="2026-11-03 17:30:03.743733281 +0000 UTC m=+4499.076946854" Nov 03 17:30:03.823343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.823285 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" podStartSLOduration=3.823269066 podStartE2EDuration="3.823269066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.821089966 +0000 UTC m=+4499.154303553" watchObservedRunningTime="2026-11-03 17:30:03.823269066 +0000 UTC m=+4499.156482634" Nov 03 17:30:03.864556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.864416 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" podStartSLOduration=3.8644037129999997 podStartE2EDuration="3.864403713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.863054595 +0000 UTC m=+4499.196268167" watchObservedRunningTime="2026-11-03 17:30:03.864403713 +0000 UTC m=+4499.197617282" Nov 03 17:30:03.907806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:03.907754 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" podStartSLOduration=3.907738844 podStartE2EDuration="3.907738844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:03.90436984 +0000 UTC m=+4499.237583410" watchObservedRunningTime="2026-11-03 17:30:03.907738844 +0000 UTC m=+4499.240952412" Nov 03 17:30:04.514372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.514331 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerStarted","Data":"3f582bb13c129561e83c97c0f7b4a96e4d3c9bec081f853096a392b18cb7e5dd"} Nov 03 17:30:04.516223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.516195 6001 generic.go:356] "Generic (PLEG): container finished" podID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerID="6a4a6c0a13725f96d6b80c381f106aad1b77d16ee149654c950946efb90bcefe" exitCode=60 Nov 03 17:30:04.516360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.516274 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerDied","Data":"6a4a6c0a13725f96d6b80c381f106aad1b77d16ee149654c950946efb90bcefe"} Nov 03 17:30:04.516604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.516586 6001 scope.go:122] "RemoveContainer" containerID="6a4a6c0a13725f96d6b80c381f106aad1b77d16ee149654c950946efb90bcefe" Nov 03 17:30:04.518512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.518490 6001 generic.go:356] "Generic (PLEG): container finished" podID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerID="8aedc9e9664ac29c7fdd508950b3c632f39718b8b44aef11991c952b39fba065" exitCode=60 Nov 03 17:30:04.518621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.518582 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerDied","Data":"8aedc9e9664ac29c7fdd508950b3c632f39718b8b44aef11991c952b39fba065"} Nov 03 17:30:04.518849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.518829 6001 scope.go:122] "RemoveContainer" containerID="8aedc9e9664ac29c7fdd508950b3c632f39718b8b44aef11991c952b39fba065" Nov 03 17:30:04.520660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.520630 6001 generic.go:356] "Generic (PLEG): container finished" podID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerID="a5c45aae9f8728c80496a1ecbdde2b5498692dbe3ddd2842ad9b8ef0b6597aac" exitCode=60 Nov 03 17:30:04.520786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.520665 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerDied","Data":"a5c45aae9f8728c80496a1ecbdde2b5498692dbe3ddd2842ad9b8ef0b6597aac"} Nov 03 17:30:04.520986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.520969 6001 scope.go:122] "RemoveContainer" containerID="a5c45aae9f8728c80496a1ecbdde2b5498692dbe3ddd2842ad9b8ef0b6597aac" Nov 03 17:30:04.523064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.523037 6001 generic.go:356] "Generic (PLEG): container finished" podID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerID="06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8" exitCode=60 Nov 03 17:30:04.523167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.523123 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerDied","Data":"06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8"} Nov 03 17:30:04.523225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.523173 6001 scope.go:122] "RemoveContainer" containerID="3df41bdb6d293d3b9f1872a672a4c59e1d25776a7c17d3d5ef106adbce700b26" Nov 03 17:30:04.523427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.523404 6001 scope.go:122] "RemoveContainer" containerID="06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8" Nov 03 17:30:04.523753 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:04.523722 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll_odh-ai-gateway-infra(b02a0e58-8cb5-4057-ac74-4b52d64f3176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" Nov 03 17:30:04.524904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.524883 6001 generic.go:356] "Generic (PLEG): container finished" podID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerID="9fa623ddd32e4691b6200272a96132ef3373f39f2595d40cf50a97d369079a96" exitCode=60 Nov 03 17:30:04.525010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.524907 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerDied","Data":"9fa623ddd32e4691b6200272a96132ef3373f39f2595d40cf50a97d369079a96"} Nov 03 17:30:04.525199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.525184 6001 scope.go:122] "RemoveContainer" containerID="9fa623ddd32e4691b6200272a96132ef3373f39f2595d40cf50a97d369079a96" Nov 03 17:30:04.526919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.526900 6001 generic.go:356] "Generic (PLEG): container finished" podID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerID="156ebe1bad659e0df8c9d8d27ae4096c7da1ef316fc3537463759147ff572969" exitCode=60 Nov 03 17:30:04.527032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.526997 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerDied","Data":"156ebe1bad659e0df8c9d8d27ae4096c7da1ef316fc3537463759147ff572969"} Nov 03 17:30:04.527283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.527267 6001 scope.go:122] "RemoveContainer" containerID="156ebe1bad659e0df8c9d8d27ae4096c7da1ef316fc3537463759147ff572969" Nov 03 17:30:04.528753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.528734 6001 generic.go:356] "Generic (PLEG): container finished" podID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerID="64beab449167725e49248ea05f6ff86437f2e23239e1725e6b8c32957df70d01" exitCode=60 Nov 03 17:30:04.528860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.528805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerDied","Data":"64beab449167725e49248ea05f6ff86437f2e23239e1725e6b8c32957df70d01"} Nov 03 17:30:04.529012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.528999 6001 scope.go:122] "RemoveContainer" containerID="64beab449167725e49248ea05f6ff86437f2e23239e1725e6b8c32957df70d01" Nov 03 17:30:04.530729 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.530707 6001 generic.go:356] "Generic (PLEG): container finished" podID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerID="0e4b0ac443b2982e53532481d4a396a20c78b865017c1c97606f16aaef32aecd" exitCode=60 Nov 03 17:30:04.530833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.530746 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerDied","Data":"0e4b0ac443b2982e53532481d4a396a20c78b865017c1c97606f16aaef32aecd"} Nov 03 17:30:04.531019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.530997 6001 scope.go:122] "RemoveContainer" containerID="0e4b0ac443b2982e53532481d4a396a20c78b865017c1c97606f16aaef32aecd" Nov 03 17:30:04.534121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.534100 6001 generic.go:356] "Generic (PLEG): container finished" podID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerID="5ec8956f3dacd9d338b0a0b0abc8952e320bf75c340aad903c14e1d02a43a147" exitCode=60 Nov 03 17:30:04.534726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.534575 6001 scope.go:122] "RemoveContainer" containerID="5ec8956f3dacd9d338b0a0b0abc8952e320bf75c340aad903c14e1d02a43a147" Nov 03 17:30:04.534726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.534607 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerDied","Data":"5ec8956f3dacd9d338b0a0b0abc8952e320bf75c340aad903c14e1d02a43a147"} Nov 03 17:30:04.536676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.536436 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerID="4eb1a33adbc59e59290726038cf66663994a429638b2fb46326926c4f87ce61b" exitCode=60 Nov 03 17:30:04.536676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.536476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerDied","Data":"4eb1a33adbc59e59290726038cf66663994a429638b2fb46326926c4f87ce61b"} Nov 03 17:30:04.536793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.536776 6001 scope.go:122] "RemoveContainer" containerID="4eb1a33adbc59e59290726038cf66663994a429638b2fb46326926c4f87ce61b" Nov 03 17:30:04.538071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.538052 6001 generic.go:356] "Generic (PLEG): container finished" podID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerID="3ed8c41bf9c7cb0136291898c47b456a6ff92c5f3f0ab29264124f98c0ec36c6" exitCode=60 Nov 03 17:30:04.538145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.538128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerDied","Data":"3ed8c41bf9c7cb0136291898c47b456a6ff92c5f3f0ab29264124f98c0ec36c6"} Nov 03 17:30:04.538352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.538333 6001 scope.go:122] "RemoveContainer" containerID="3ed8c41bf9c7cb0136291898c47b456a6ff92c5f3f0ab29264124f98c0ec36c6" Nov 03 17:30:04.539938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.539916 6001 generic.go:356] "Generic (PLEG): container finished" podID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerID="161e000bdf06d5c5b67a2dd97a08a6e168ef72615c04350f1a6c00f840c99792" exitCode=60 Nov 03 17:30:04.540040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.539953 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerDied","Data":"161e000bdf06d5c5b67a2dd97a08a6e168ef72615c04350f1a6c00f840c99792"} Nov 03 17:30:04.540176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.540163 6001 scope.go:122] "RemoveContainer" containerID="161e000bdf06d5c5b67a2dd97a08a6e168ef72615c04350f1a6c00f840c99792" Nov 03 17:30:04.556043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:04.555923 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" podStartSLOduration=4.555905603 podStartE2EDuration="4.555905603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:04.555683194 +0000 UTC m=+4499.888896767" watchObservedRunningTime="2026-11-03 17:30:04.555905603 +0000 UTC m=+4499.889119174" Nov 03 17:30:05.566187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.566092 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerStarted","Data":"51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9"} Nov 03 17:30:05.569697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.569664 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerStarted","Data":"317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98"} Nov 03 17:30:05.572775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.572754 6001 scope.go:122] "RemoveContainer" containerID="06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8" Nov 03 17:30:05.573161 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:05.573135 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll_odh-ai-gateway-infra(b02a0e58-8cb5-4057-ac74-4b52d64f3176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" Nov 03 17:30:05.574768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.574739 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerStarted","Data":"a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129"} Nov 03 17:30:05.576699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.576677 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerStarted","Data":"3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1"} Nov 03 17:30:05.578688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.578665 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerStarted","Data":"7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269"} Nov 03 17:30:05.580809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.580784 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerStarted","Data":"6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433"} Nov 03 17:30:05.583061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.583034 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerStarted","Data":"9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d"} Nov 03 17:30:05.584789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.584767 6001 generic.go:356] "Generic (PLEG): container finished" podID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerID="3f582bb13c129561e83c97c0f7b4a96e4d3c9bec081f853096a392b18cb7e5dd" exitCode=60 Nov 03 17:30:05.584907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.584840 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerDied","Data":"3f582bb13c129561e83c97c0f7b4a96e4d3c9bec081f853096a392b18cb7e5dd"} Nov 03 17:30:05.585059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.585041 6001 scope.go:122] "RemoveContainer" containerID="3f582bb13c129561e83c97c0f7b4a96e4d3c9bec081f853096a392b18cb7e5dd" Nov 03 17:30:05.586815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.586791 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerStarted","Data":"24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558"} Nov 03 17:30:05.611492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.611412 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" podStartSLOduration=5.611392103 podStartE2EDuration="5.611392103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:05.609192817 +0000 UTC m=+4500.942406407" watchObservedRunningTime="2026-11-03 17:30:05.611392103 +0000 UTC m=+4500.944605673" Nov 03 17:30:05.765154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.765019 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:30:05.787557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.787507 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" podStartSLOduration=5.787493682 podStartE2EDuration="5.787493682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:05.782539 +0000 UTC m=+4501.115752573" watchObservedRunningTime="2026-11-03 17:30:05.787493682 +0000 UTC m=+4501.120707250" Nov 03 17:30:05.881759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.881731 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:30:05.914243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.914178 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" podStartSLOduration=5.914158852 podStartE2EDuration="5.914158852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:05.91007138 +0000 UTC m=+4501.243284951" watchObservedRunningTime="2026-11-03 17:30:05.914158852 +0000 UTC m=+4501.247372421" Nov 03 17:30:05.948678 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:05.943557 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47398737_8736_4f4b_b8d4_9ba4d716ea06.slice/crio-6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:30:05.963144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:05.961461 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" podStartSLOduration=5.961422232 podStartE2EDuration="5.961422232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:05.951813702 +0000 UTC m=+4501.285027271" watchObservedRunningTime="2026-11-03 17:30:05.961422232 +0000 UTC m=+4501.294635801" Nov 03 17:30:06.598484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.598430 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerID="9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d" exitCode=60 Nov 03 17:30:06.598484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.598477 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerDied","Data":"9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d"} Nov 03 17:30:06.598997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.598565 6001 scope.go:122] "RemoveContainer" containerID="4eb1a33adbc59e59290726038cf66663994a429638b2fb46326926c4f87ce61b" Nov 03 17:30:06.598997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.598865 6001 scope.go:122] "RemoveContainer" containerID="9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d" Nov 03 17:30:06.599163 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.599142 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4_odh-ai-gateway-infra(3fa32e18-1a18-43ea-be00-01871c323b4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" Nov 03 17:30:06.600919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.600896 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerStarted","Data":"c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc"} Nov 03 17:30:06.603044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.603017 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerStarted","Data":"b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7"} Nov 03 17:30:06.605375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.605338 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerStarted","Data":"e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f"} Nov 03 17:30:06.607583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.607557 6001 generic.go:356] "Generic (PLEG): container finished" podID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerID="24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558" exitCode=60 Nov 03 17:30:06.607694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.607638 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerDied","Data":"24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558"} Nov 03 17:30:06.607919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.607899 6001 scope.go:122] "RemoveContainer" containerID="24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558" Nov 03 17:30:06.608211 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.608191 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48_odh-ai-gateway-infra(0fdcae68-e0ee-4f64-9482-a5da2b888a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" Nov 03 17:30:06.609689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.609669 6001 generic.go:356] "Generic (PLEG): container finished" podID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerID="51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9" exitCode=60 Nov 03 17:30:06.609784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.609745 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerDied","Data":"51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9"} Nov 03 17:30:06.609962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.609947 6001 scope.go:122] "RemoveContainer" containerID="51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9" Nov 03 17:30:06.610231 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.610205 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25_odh-ai-gateway-infra(46fac7a2-4eda-4e5c-be12-531425e2e807)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" Nov 03 17:30:06.611740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.611719 6001 generic.go:356] "Generic (PLEG): container finished" podID="426d8922-d11b-4983-8b2d-52467d60dade" containerID="317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98" exitCode=60 Nov 03 17:30:06.611825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.611794 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerDied","Data":"317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98"} Nov 03 17:30:06.611967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.611954 6001 scope.go:122] "RemoveContainer" containerID="317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98" Nov 03 17:30:06.612216 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.612194 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc_odh-ai-gateway-infra(426d8922-d11b-4983-8b2d-52467d60dade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" podUID="426d8922-d11b-4983-8b2d-52467d60dade" Nov 03 17:30:06.613730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.613709 6001 generic.go:356] "Generic (PLEG): container finished" podID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerID="050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816" exitCode=60 Nov 03 17:30:06.613845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.613788 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerDied","Data":"050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816"} Nov 03 17:30:06.614082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.614064 6001 scope.go:122] "RemoveContainer" containerID="050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816" Nov 03 17:30:06.614345 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.614324 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq_odh-ai-gateway-infra(cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" Nov 03 17:30:06.616102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.616081 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerStarted","Data":"183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b"} Nov 03 17:30:06.618059 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.618035 6001 generic.go:356] "Generic (PLEG): container finished" podID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerID="a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129" exitCode=60 Nov 03 17:30:06.618153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.618101 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerDied","Data":"a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129"} Nov 03 17:30:06.618328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.618311 6001 scope.go:122] "RemoveContainer" containerID="a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129" Nov 03 17:30:06.618624 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.618593 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc_odh-ai-gateway-infra(02617808-3d0b-4cbb-ac17-ec67fbac6ae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" Nov 03 17:30:06.620125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.620106 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerStarted","Data":"22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648"} Nov 03 17:30:06.622098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.622076 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerStarted","Data":"afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475"} Nov 03 17:30:06.624128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.624105 6001 generic.go:356] "Generic (PLEG): container finished" podID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerID="9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d" exitCode=60 Nov 03 17:30:06.624240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.624178 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerDied","Data":"9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d"} Nov 03 17:30:06.624485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.624465 6001 scope.go:122] "RemoveContainer" containerID="9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d" Nov 03 17:30:06.624743 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.624722 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44_odh-ai-gateway-infra(c1c917b6-5e0b-4235-85a3-55acd85f2f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" Nov 03 17:30:06.626277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.626260 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerStarted","Data":"3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe"} Nov 03 17:30:06.628267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.628249 6001 generic.go:356] "Generic (PLEG): container finished" podID="8565be49-1874-41cd-b659-aa5a09a24c74" containerID="3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1" exitCode=60 Nov 03 17:30:06.628379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.628329 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerDied","Data":"3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1"} Nov 03 17:30:06.628557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.628540 6001 scope.go:122] "RemoveContainer" containerID="3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1" Nov 03 17:30:06.628777 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.628758 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh_odh-ai-gateway-infra(8565be49-1874-41cd-b659-aa5a09a24c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" Nov 03 17:30:06.630311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.630291 6001 generic.go:356] "Generic (PLEG): container finished" podID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerID="7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269" exitCode=60 Nov 03 17:30:06.630408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.630369 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerDied","Data":"7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269"} Nov 03 17:30:06.631525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.630581 6001 scope.go:122] "RemoveContainer" containerID="7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269" Nov 03 17:30:06.631525 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.630816 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z_odh-ai-gateway-infra(32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" Nov 03 17:30:06.632502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.632484 6001 generic.go:356] "Generic (PLEG): container finished" podID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerID="6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433" exitCode=60 Nov 03 17:30:06.632591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.632517 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerDied","Data":"6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433"} Nov 03 17:30:06.632785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.632773 6001 scope.go:122] "RemoveContainer" containerID="6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433" Nov 03 17:30:06.632995 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:06.632978 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq_odh-ai-gateway-infra(47398737-8736-4f4b-b8d4-9ba4d716ea06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" Nov 03 17:30:06.880961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:06.880902 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" podStartSLOduration=6.880886416 podStartE2EDuration="6.880886416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:06.880143935 +0000 UTC m=+4502.213357505" watchObservedRunningTime="2026-11-03 17:30:06.880886416 +0000 UTC m=+4502.214099985" Nov 03 17:30:07.103002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.102847 6001 scope.go:122] "RemoveContainer" containerID="6a4a6c0a13725f96d6b80c381f106aad1b77d16ee149654c950946efb90bcefe" Nov 03 17:30:07.118801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.118778 6001 scope.go:122] "RemoveContainer" containerID="8aedc9e9664ac29c7fdd508950b3c632f39718b8b44aef11991c952b39fba065" Nov 03 17:30:07.137356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.137332 6001 scope.go:122] "RemoveContainer" containerID="0d0148c4e908e00b2ada9eb020b5b206ffb8b73c88ef3dba4c34ee6aa38ec41a" Nov 03 17:30:07.154107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.154085 6001 scope.go:122] "RemoveContainer" containerID="a5c45aae9f8728c80496a1ecbdde2b5498692dbe3ddd2842ad9b8ef0b6597aac" Nov 03 17:30:07.170501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.170436 6001 scope.go:122] "RemoveContainer" containerID="5401d0c22d69ee333f8305ba0b2e84fa3fc3a358f91047bfcae94f9f10e04451" Nov 03 17:30:07.187393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.187364 6001 scope.go:122] "RemoveContainer" containerID="0e4b0ac443b2982e53532481d4a396a20c78b865017c1c97606f16aaef32aecd" Nov 03 17:30:07.205281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.205255 6001 scope.go:122] "RemoveContainer" containerID="466c3aeeaf58af5975ae90873e6657cef8b6c3b2d7183689d7845dece46a2848" Nov 03 17:30:07.221470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.221429 6001 scope.go:122] "RemoveContainer" containerID="75d670683578a3547d8556177a350dd02eaaf7f47634a94343204201f96b2676" Nov 03 17:30:07.242180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.242156 6001 scope.go:122] "RemoveContainer" containerID="548ab20db5cfc0258f1a95f598e4b9c0a0c0e67c3d2e99da51af71c1764847cf" Nov 03 17:30:07.643666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.643633 6001 scope.go:122] "RemoveContainer" containerID="9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d" Nov 03 17:30:07.644135 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.644078 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4_odh-ai-gateway-infra(3fa32e18-1a18-43ea-be00-01871c323b4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" Nov 03 17:30:07.645246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.645219 6001 generic.go:356] "Generic (PLEG): container finished" podID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerID="c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc" exitCode=60 Nov 03 17:30:07.645376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.645304 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerDied","Data":"c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc"} Nov 03 17:30:07.645376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.645363 6001 scope.go:122] "RemoveContainer" containerID="3ed8c41bf9c7cb0136291898c47b456a6ff92c5f3f0ab29264124f98c0ec36c6" Nov 03 17:30:07.645635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.645609 6001 scope.go:122] "RemoveContainer" containerID="c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc" Nov 03 17:30:07.645971 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.645946 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp_odh-ai-gateway-infra(e1547f64-57e7-4641-bee9-c1449c90b12f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" Nov 03 17:30:07.647697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.647672 6001 generic.go:356] "Generic (PLEG): container finished" podID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerID="b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7" exitCode=60 Nov 03 17:30:07.647797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.647745 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerDied","Data":"b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7"} Nov 03 17:30:07.647998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.647982 6001 scope.go:122] "RemoveContainer" containerID="b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7" Nov 03 17:30:07.648319 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.648300 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg_odh-ai-gateway-infra(d7d0bf64-0564-4905-8d4d-b932579d9f69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" Nov 03 17:30:07.649696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.649676 6001 generic.go:356] "Generic (PLEG): container finished" podID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerID="e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f" exitCode=60 Nov 03 17:30:07.649781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.649752 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerDied","Data":"e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f"} Nov 03 17:30:07.649995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.649980 6001 scope.go:122] "RemoveContainer" containerID="e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f" Nov 03 17:30:07.650308 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.650291 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql_odh-ai-gateway-infra(a6a4ade1-ce40-48f5-8b65-94a72868c47a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" Nov 03 17:30:07.651881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.651859 6001 scope.go:122] "RemoveContainer" containerID="24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558" Nov 03 17:30:07.652201 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.652182 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48_odh-ai-gateway-infra(0fdcae68-e0ee-4f64-9482-a5da2b888a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" Nov 03 17:30:07.653524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.653504 6001 scope.go:122] "RemoveContainer" containerID="51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9" Nov 03 17:30:07.653849 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.653828 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25_odh-ai-gateway-infra(46fac7a2-4eda-4e5c-be12-531425e2e807)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" Nov 03 17:30:07.655136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.655120 6001 scope.go:122] "RemoveContainer" containerID="317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98" Nov 03 17:30:07.655454 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.655415 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc_odh-ai-gateway-infra(426d8922-d11b-4983-8b2d-52467d60dade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" podUID="426d8922-d11b-4983-8b2d-52467d60dade" Nov 03 17:30:07.656763 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.656743 6001 scope.go:122] "RemoveContainer" containerID="050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816" Nov 03 17:30:07.657005 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.656984 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq_odh-ai-gateway-infra(cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" Nov 03 17:30:07.658817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.658796 6001 generic.go:356] "Generic (PLEG): container finished" podID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerID="183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b" exitCode=60 Nov 03 17:30:07.658925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.658841 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerDied","Data":"183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b"} Nov 03 17:30:07.659138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.659123 6001 scope.go:122] "RemoveContainer" containerID="183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b" Nov 03 17:30:07.659418 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.659399 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr_odh-ai-gateway-infra(1906ff6e-c536-416b-a1e6-c1995cc23c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" Nov 03 17:30:07.660771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.660755 6001 scope.go:122] "RemoveContainer" containerID="a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129" Nov 03 17:30:07.661053 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.661027 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc_odh-ai-gateway-infra(02617808-3d0b-4cbb-ac17-ec67fbac6ae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" Nov 03 17:30:07.662740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.662717 6001 generic.go:356] "Generic (PLEG): container finished" podID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerID="22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648" exitCode=60 Nov 03 17:30:07.662826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.662791 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerDied","Data":"22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648"} Nov 03 17:30:07.663003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.662987 6001 scope.go:122] "RemoveContainer" containerID="22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648" Nov 03 17:30:07.663242 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.663220 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d_odh-ai-gateway-infra(dddbd6bf-5f2b-47a6-90c6-85daf77b6839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" Nov 03 17:30:07.664103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.664088 6001 scope.go:122] "RemoveContainer" containerID="161e000bdf06d5c5b67a2dd97a08a6e168ef72615c04350f1a6c00f840c99792" Nov 03 17:30:07.664801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.664777 6001 generic.go:356] "Generic (PLEG): container finished" podID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerID="afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475" exitCode=60 Nov 03 17:30:07.664886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.664853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerDied","Data":"afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475"} Nov 03 17:30:07.665022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.665009 6001 scope.go:122] "RemoveContainer" containerID="afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475" Nov 03 17:30:07.665222 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.665206 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt_odh-ai-gateway-infra(e4d7b1e1-09a5-4273-9cc0-2a278231ddf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" Nov 03 17:30:07.666860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.666844 6001 scope.go:122] "RemoveContainer" containerID="9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d" Nov 03 17:30:07.667114 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.667091 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44_odh-ai-gateway-infra(c1c917b6-5e0b-4235-85a3-55acd85f2f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" Nov 03 17:30:07.668434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.668417 6001 generic.go:356] "Generic (PLEG): container finished" podID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerID="3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe" exitCode=60 Nov 03 17:30:07.668524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.668472 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerDied","Data":"3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe"} Nov 03 17:30:07.668702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.668688 6001 scope.go:122] "RemoveContainer" containerID="3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe" Nov 03 17:30:07.668911 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.668893 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r_odh-ai-gateway-infra(09e24410-4d9e-4116-93b5-aa72aac5672e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" Nov 03 17:30:07.670407 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.670389 6001 scope.go:122] "RemoveContainer" containerID="3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1" Nov 03 17:30:07.670720 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.670693 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh_odh-ai-gateway-infra(8565be49-1874-41cd-b659-aa5a09a24c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" Nov 03 17:30:07.672411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.672394 6001 scope.go:122] "RemoveContainer" containerID="7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269" Nov 03 17:30:07.672634 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.672613 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z_odh-ai-gateway-infra(32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" Nov 03 17:30:07.674196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.674180 6001 scope.go:122] "RemoveContainer" containerID="6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433" Nov 03 17:30:07.674387 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:07.674370 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq_odh-ai-gateway-infra(47398737-8736-4f4b-b8d4-9ba4d716ea06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" Nov 03 17:30:07.680150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.680133 6001 scope.go:122] "RemoveContainer" containerID="3f582bb13c129561e83c97c0f7b4a96e4d3c9bec081f853096a392b18cb7e5dd" Nov 03 17:30:07.696337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.696318 6001 scope.go:122] "RemoveContainer" containerID="9fa623ddd32e4691b6200272a96132ef3373f39f2595d40cf50a97d369079a96" Nov 03 17:30:07.711096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.711075 6001 scope.go:122] "RemoveContainer" containerID="156ebe1bad659e0df8c9d8d27ae4096c7da1ef316fc3537463759147ff572969" Nov 03 17:30:07.725808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.725783 6001 scope.go:122] "RemoveContainer" containerID="64beab449167725e49248ea05f6ff86437f2e23239e1725e6b8c32957df70d01" Nov 03 17:30:07.740662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:07.740638 6001 scope.go:122] "RemoveContainer" containerID="5ec8956f3dacd9d338b0a0b0abc8952e320bf75c340aad903c14e1d02a43a147" Nov 03 17:30:08.260571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.260541 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:30:08.313990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.313965 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:30:08.340036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.339996 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\") pod \"5684e128-aa30-430f-8604-8027740756cf\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " Nov 03 17:30:08.340191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.340175 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\" (UniqueName: \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\") pod \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " Nov 03 17:30:08.340288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.340274 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\" (UniqueName: \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\") pod \"5684e128-aa30-430f-8604-8027740756cf\" (UID: \"5684e128-aa30-430f-8604-8027740756cf\") " Nov 03 17:30:08.340361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.340336 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\") pod \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\" (UID: \"7996d93d-b563-40ff-baeb-0a6e3f8ba227\") " Nov 03 17:30:08.340428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.340361 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume" pod "5684e128-aa30-430f-8604-8027740756cf" (UID: "5684e128-aa30-430f-8604-8027740756cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:08.340778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.340756 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume" pod "7996d93d-b563-40ff-baeb-0a6e3f8ba227" (UID: "7996d93d-b563-40ff-baeb-0a6e3f8ba227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:08.341058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.341039 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5684e128-aa30-430f-8604-8027740756cf-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:08.341150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.341062 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7996d93d-b563-40ff-baeb-0a6e3f8ba227-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:08.342349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.342329 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x" pod "7996d93d-b563-40ff-baeb-0a6e3f8ba227" (UID: "7996d93d-b563-40ff-baeb-0a6e3f8ba227"). InnerVolumeSpecName "kube-api-access-blk2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:08.342515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.342492 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb" pod "5684e128-aa30-430f-8604-8027740756cf" (UID: "5684e128-aa30-430f-8604-8027740756cf"). InnerVolumeSpecName "kube-api-access-9frzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:08.442625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.442578 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blk2x\" (UniqueName: \"kubernetes.io/projected/7996d93d-b563-40ff-baeb-0a6e3f8ba227-kube-api-access-blk2x\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:08.442625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.442609 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9frzb\" (UniqueName: \"kubernetes.io/projected/5684e128-aa30-430f-8604-8027740756cf-kube-api-access-9frzb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:08.690042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.690010 6001 generic.go:356] "Generic (PLEG): container finished" podID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerID="57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4" exitCode=0 Nov 03 17:30:08.690561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.690055 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" Nov 03 17:30:08.690561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.690088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" event={"ID":"7996d93d-b563-40ff-baeb-0a6e3f8ba227","Type":"ContainerDied","Data":"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4"} Nov 03 17:30:08.690561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.690168 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp" event={"ID":"7996d93d-b563-40ff-baeb-0a6e3f8ba227","Type":"ContainerDied","Data":"ae925cf709942f0b171aec964d84b542dacd0f3d0faaa77e4acafca22806725c"} Nov 03 17:30:08.690561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.690195 6001 scope.go:122] "RemoveContainer" containerID="57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4" Nov 03 17:30:08.693300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.693276 6001 scope.go:122] "RemoveContainer" containerID="c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc" Nov 03 17:30:08.693681 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.693655 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp_odh-ai-gateway-infra(e1547f64-57e7-4641-bee9-c1449c90b12f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" Nov 03 17:30:08.695199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.695177 6001 scope.go:122] "RemoveContainer" containerID="b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7" Nov 03 17:30:08.695545 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.695520 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg_odh-ai-gateway-infra(d7d0bf64-0564-4905-8d4d-b932579d9f69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" Nov 03 17:30:08.696983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.696962 6001 scope.go:122] "RemoveContainer" containerID="e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f" Nov 03 17:30:08.697241 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.697220 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql_odh-ai-gateway-infra(a6a4ade1-ce40-48f5-8b65-94a72868c47a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" Nov 03 17:30:08.698899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.698868 6001 scope.go:122] "RemoveContainer" containerID="183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b" Nov 03 17:30:08.699174 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.699143 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr_odh-ai-gateway-infra(1906ff6e-c536-416b-a1e6-c1995cc23c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" Nov 03 17:30:08.700688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.700670 6001 scope.go:122] "RemoveContainer" containerID="22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648" Nov 03 17:30:08.700947 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.700922 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d_odh-ai-gateway-infra(dddbd6bf-5f2b-47a6-90c6-85daf77b6839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" Nov 03 17:30:08.702374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.702353 6001 scope.go:122] "RemoveContainer" containerID="afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475" Nov 03 17:30:08.702641 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.702612 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt_odh-ai-gateway-infra(e4d7b1e1-09a5-4273-9cc0-2a278231ddf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" Nov 03 17:30:08.703946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.703930 6001 scope.go:122] "RemoveContainer" containerID="3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe" Nov 03 17:30:08.704158 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.704132 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r_odh-ai-gateway-infra(09e24410-4d9e-4116-93b5-aa72aac5672e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" Nov 03 17:30:08.704822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.704801 6001 scope.go:122] "RemoveContainer" containerID="57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4" Nov 03 17:30:08.705095 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.705074 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4\": container with ID starting with 57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4 not found: ID does not exist" containerID="57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4" Nov 03 17:30:08.705161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705102 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4"} err="failed to get container status \"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4\": rpc error: code = NotFound desc = could not find container \"57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4\": container with ID starting with 57cb468da7b568df380362d2b9cbd03801484cfdd9975ff18e6f3001ef67f2e4 not found: ID does not exist" Nov 03 17:30:08.705205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705175 6001 generic.go:356] "Generic (PLEG): container finished" podID="5684e128-aa30-430f-8604-8027740756cf" containerID="417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d" exitCode=0 Nov 03 17:30:08.705239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705206 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" event={"ID":"5684e128-aa30-430f-8604-8027740756cf","Type":"ContainerDied","Data":"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d"} Nov 03 17:30:08.705239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705210 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" Nov 03 17:30:08.705239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705237 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp" event={"ID":"5684e128-aa30-430f-8604-8027740756cf","Type":"ContainerDied","Data":"9829cedf4d9be05a9f6c2537bda24ed47283f6a550965942bf018099d8fbae69"} Nov 03 17:30:08.705437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.705256 6001 scope.go:122] "RemoveContainer" containerID="417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d" Nov 03 17:30:08.717113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.717090 6001 scope.go:122] "RemoveContainer" containerID="417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d" Nov 03 17:30:08.717412 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:08.717389 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d\": container with ID starting with 417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d not found: ID does not exist" containerID="417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d" Nov 03 17:30:08.717511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:08.717423 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d"} err="failed to get container status \"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d\": rpc error: code = NotFound desc = could not find container \"417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d\": container with ID starting with 417469c99e6356fe34a092a1407be81dab55e85641fd83d8163bbd0abbff753d not found: ID does not exist" Nov 03 17:30:09.071022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:09.070980 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:30:09.075355 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:09.075325 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-fdbf0c-684c66d767-gfvtp"] Nov 03 17:30:09.109050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:09.109014 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:30:09.113153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:09.113128 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-fdbf0c-86f4687bf5-qv4zp"] Nov 03 17:30:10.792235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:10.792196 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5684e128-aa30-430f-8604-8027740756cf" path="/var/lib/kubelet/pods/5684e128-aa30-430f-8604-8027740756cf/volumes" Nov 03 17:30:10.792702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:10.792687 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" path="/var/lib/kubelet/pods/7996d93d-b563-40ff-baeb-0a6e3f8ba227/volumes" Nov 03 17:30:13.289002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:13.288971 6001 scope.go:122] "RemoveContainer" containerID="f9737921c87eeaee33e7878a7d5e2e21ec178c755f4a0cf7909c0867f2c9d6a0" Nov 03 17:30:13.300502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:13.300480 6001 scope.go:122] "RemoveContainer" containerID="6c67da8a771958bb4ccd0fcde91797aa2dabfb01e71e53115b7109334d52fab3" Nov 03 17:30:13.311000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:13.310977 6001 scope.go:122] "RemoveContainer" containerID="d54278b4dceea411e01bda407fb7a7a65a7e14240552c703705725ca94f9670f" Nov 03 17:30:13.326351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:13.326313 6001 scope.go:122] "RemoveContainer" containerID="b36b7839420975c9d266550d65f75eebe9bcc02605b182c2492af1cbdf4e275a" Nov 03 17:30:17.782408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.782317 6001 scope.go:122] "RemoveContainer" containerID="51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9" Nov 03 17:30:17.782902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.782538 6001 scope.go:122] "RemoveContainer" containerID="183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b" Nov 03 17:30:17.782902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.782679 6001 scope.go:122] "RemoveContainer" containerID="22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648" Nov 03 17:30:17.782902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.782847 6001 scope.go:122] "RemoveContainer" containerID="7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269" Nov 03 17:30:17.784248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.783201 6001 scope.go:122] "RemoveContainer" containerID="a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129" Nov 03 17:30:17.784248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.783427 6001 scope.go:122] "RemoveContainer" containerID="afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475" Nov 03 17:30:17.784248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.783532 6001 scope.go:122] "RemoveContainer" containerID="6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433" Nov 03 17:30:17.784248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:17.783865 6001 scope.go:122] "RemoveContainer" containerID="317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98" Nov 03 17:30:18.032508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.032391 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6"] Nov 03 17:30:18.044334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.044296 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b586ec8c-t8gl6"] Nov 03 17:30:18.781835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.781790 6001 scope.go:122] "RemoveContainer" containerID="c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc" Nov 03 17:30:18.782160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.781946 6001 scope.go:122] "RemoveContainer" containerID="06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8" Nov 03 17:30:18.793267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.793241 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88369823-5e2d-4a70-ae36-83b1d47d5114" path="/var/lib/kubelet/pods/88369823-5e2d-4a70-ae36-83b1d47d5114/volumes" Nov 03 17:30:18.846773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.845395 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerStarted","Data":"7d3374b39b25f206eaa917ca1a03e61951bd006eed868a64623beec09993ab6e"} Nov 03 17:30:18.852870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.852831 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerStarted","Data":"f3c36aa5b4097bf8ab633fe23d5c24237e5d10aafe724c23099ddf9fe86c4d84"} Nov 03 17:30:18.855660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.855628 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerStarted","Data":"153e62ae24b1a2f43923a3b63ab4f2ec48c4711c144db3ba083a8dd41f89fa58"} Nov 03 17:30:18.857836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.857807 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerStarted","Data":"18b681211a05ea2c8c3440dbae5b9526779ab9e146bd4dff7718cbc10ff696b0"} Nov 03 17:30:18.859874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.859846 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerStarted","Data":"56b0ffb09f6911a6e4a66c106c3c0ace7264d9e0d61c0381d1c4b587329ab729"} Nov 03 17:30:18.862025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.862003 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerStarted","Data":"adeb0532abf4ebac66b2ae096dc8cc701121eca18df09b7ff51b56bb53ff622e"} Nov 03 17:30:18.864313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.864279 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerStarted","Data":"95a5ab97669799396f68506549dc12adbc43ac1e7f6db60c121d09037c6d38b7"} Nov 03 17:30:18.866872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:18.866846 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerStarted","Data":"e6fe56983038bf2035a2c90c96dca35213e5000042431a06125ae14810bb2eb2"} Nov 03 17:30:19.782588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.782549 6001 scope.go:122] "RemoveContainer" containerID="3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1" Nov 03 17:30:19.782986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.782960 6001 scope.go:122] "RemoveContainer" containerID="24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558" Nov 03 17:30:19.783123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.783027 6001 scope.go:122] "RemoveContainer" containerID="e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f" Nov 03 17:30:19.878910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.878873 6001 generic.go:356] "Generic (PLEG): container finished" podID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerID="adeb0532abf4ebac66b2ae096dc8cc701121eca18df09b7ff51b56bb53ff622e" exitCode=60 Nov 03 17:30:19.879369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.878939 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerDied","Data":"adeb0532abf4ebac66b2ae096dc8cc701121eca18df09b7ff51b56bb53ff622e"} Nov 03 17:30:19.879369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.879000 6001 scope.go:122] "RemoveContainer" containerID="afc372d300cbf3c2cf169bb1743fbbbab44414f2303d329c2c2b2c78e0eab475" Nov 03 17:30:19.879552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.879534 6001 scope.go:122] "RemoveContainer" containerID="adeb0532abf4ebac66b2ae096dc8cc701121eca18df09b7ff51b56bb53ff622e" Nov 03 17:30:19.880036 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.879987 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt_odh-ai-gateway-infra(e4d7b1e1-09a5-4273-9cc0-2a278231ddf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" Nov 03 17:30:19.881419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.881396 6001 generic.go:356] "Generic (PLEG): container finished" podID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerID="95a5ab97669799396f68506549dc12adbc43ac1e7f6db60c121d09037c6d38b7" exitCode=60 Nov 03 17:30:19.881582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.881497 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerDied","Data":"95a5ab97669799396f68506549dc12adbc43ac1e7f6db60c121d09037c6d38b7"} Nov 03 17:30:19.881981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.881959 6001 scope.go:122] "RemoveContainer" containerID="95a5ab97669799396f68506549dc12adbc43ac1e7f6db60c121d09037c6d38b7" Nov 03 17:30:19.882392 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.882357 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z_odh-ai-gateway-infra(32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" Nov 03 17:30:19.883602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.883582 6001 generic.go:356] "Generic (PLEG): container finished" podID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerID="e6fe56983038bf2035a2c90c96dca35213e5000042431a06125ae14810bb2eb2" exitCode=60 Nov 03 17:30:19.883692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.883648 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerDied","Data":"e6fe56983038bf2035a2c90c96dca35213e5000042431a06125ae14810bb2eb2"} Nov 03 17:30:19.884037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.884014 6001 scope.go:122] "RemoveContainer" containerID="e6fe56983038bf2035a2c90c96dca35213e5000042431a06125ae14810bb2eb2" Nov 03 17:30:19.884361 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.884338 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq_odh-ai-gateway-infra(47398737-8736-4f4b-b8d4-9ba4d716ea06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" Nov 03 17:30:19.886074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.886051 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerStarted","Data":"beb07957d91a2506cd8b4e36d91f7aee98dbca8e7b82fc24d62201276efc37f9"} Nov 03 17:30:19.888125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.888105 6001 generic.go:356] "Generic (PLEG): container finished" podID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerID="7d3374b39b25f206eaa917ca1a03e61951bd006eed868a64623beec09993ab6e" exitCode=60 Nov 03 17:30:19.888239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.888177 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerDied","Data":"7d3374b39b25f206eaa917ca1a03e61951bd006eed868a64623beec09993ab6e"} Nov 03 17:30:19.888548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.888530 6001 scope.go:122] "RemoveContainer" containerID="7d3374b39b25f206eaa917ca1a03e61951bd006eed868a64623beec09993ab6e" Nov 03 17:30:19.888832 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.888810 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25_odh-ai-gateway-infra(46fac7a2-4eda-4e5c-be12-531425e2e807)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" Nov 03 17:30:19.890083 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.890064 6001 generic.go:356] "Generic (PLEG): container finished" podID="426d8922-d11b-4983-8b2d-52467d60dade" containerID="f3c36aa5b4097bf8ab633fe23d5c24237e5d10aafe724c23099ddf9fe86c4d84" exitCode=60 Nov 03 17:30:19.890156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.890136 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerDied","Data":"f3c36aa5b4097bf8ab633fe23d5c24237e5d10aafe724c23099ddf9fe86c4d84"} Nov 03 17:30:19.890435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.890417 6001 scope.go:122] "RemoveContainer" containerID="f3c36aa5b4097bf8ab633fe23d5c24237e5d10aafe724c23099ddf9fe86c4d84" Nov 03 17:30:19.890755 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.890732 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc_odh-ai-gateway-infra(426d8922-d11b-4983-8b2d-52467d60dade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" podUID="426d8922-d11b-4983-8b2d-52467d60dade" Nov 03 17:30:19.891989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.891972 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerStarted","Data":"661efc795283e38c004e42860c4b8a65f941ee4e6f8df5a1fee37c5e05b6e597"} Nov 03 17:30:19.893859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.893839 6001 generic.go:356] "Generic (PLEG): container finished" podID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerID="153e62ae24b1a2f43923a3b63ab4f2ec48c4711c144db3ba083a8dd41f89fa58" exitCode=60 Nov 03 17:30:19.893974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.893907 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerDied","Data":"153e62ae24b1a2f43923a3b63ab4f2ec48c4711c144db3ba083a8dd41f89fa58"} Nov 03 17:30:19.894250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.894228 6001 scope.go:122] "RemoveContainer" containerID="153e62ae24b1a2f43923a3b63ab4f2ec48c4711c144db3ba083a8dd41f89fa58" Nov 03 17:30:19.894526 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.894504 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr_odh-ai-gateway-infra(1906ff6e-c536-416b-a1e6-c1995cc23c6a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" Nov 03 17:30:19.896415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.896383 6001 generic.go:356] "Generic (PLEG): container finished" podID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerID="18b681211a05ea2c8c3440dbae5b9526779ab9e146bd4dff7718cbc10ff696b0" exitCode=60 Nov 03 17:30:19.896552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.896474 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerDied","Data":"18b681211a05ea2c8c3440dbae5b9526779ab9e146bd4dff7718cbc10ff696b0"} Nov 03 17:30:19.896836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.896818 6001 scope.go:122] "RemoveContainer" containerID="18b681211a05ea2c8c3440dbae5b9526779ab9e146bd4dff7718cbc10ff696b0" Nov 03 17:30:19.897254 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.897110 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc_odh-ai-gateway-infra(02617808-3d0b-4cbb-ac17-ec67fbac6ae1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" Nov 03 17:30:19.898973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.898951 6001 generic.go:356] "Generic (PLEG): container finished" podID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerID="56b0ffb09f6911a6e4a66c106c3c0ace7264d9e0d61c0381d1c4b587329ab729" exitCode=60 Nov 03 17:30:19.899085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.898987 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerDied","Data":"56b0ffb09f6911a6e4a66c106c3c0ace7264d9e0d61c0381d1c4b587329ab729"} Nov 03 17:30:19.899368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.899348 6001 scope.go:122] "RemoveContainer" containerID="56b0ffb09f6911a6e4a66c106c3c0ace7264d9e0d61c0381d1c4b587329ab729" Nov 03 17:30:19.899703 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:19.899682 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d_odh-ai-gateway-infra(dddbd6bf-5f2b-47a6-90c6-85daf77b6839)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" Nov 03 17:30:19.901716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.901693 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:19.970094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.970059 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:19.976101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.976075 6001 scope.go:122] "RemoveContainer" containerID="7164b793c76308ae38e41b766db318fea04591a9658f98de4711f41a07d0d269" Nov 03 17:30:19.997893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:19.997874 6001 scope.go:122] "RemoveContainer" containerID="6eae5a809e8e7d515f8386080222ac4da4282bd148755b68007837125cad1433" Nov 03 17:30:20.008792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.008762 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:20.015677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.015600 6001 scope.go:122] "RemoveContainer" containerID="51558536f1b56d56178b870e42d8c5592aeb9affe2b88326d382ad61bf2191a9" Nov 03 17:30:20.033765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.033743 6001 scope.go:122] "RemoveContainer" containerID="317946d178e5766aae01a8d2e0d30c42dde469a624d9865982d03b4b063c9c98" Nov 03 17:30:20.056725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.056699 6001 scope.go:122] "RemoveContainer" containerID="183e3f890fafcdc805a16e18139cf265e82b264891a3f8be9e3bb1bc5117bb5b" Nov 03 17:30:20.058145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.058105 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:20.077587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.077559 6001 scope.go:122] "RemoveContainer" containerID="a3bcf7b8fa793655afb7c901edc82b88d654bb9b62e6570d1c6c1b1e5ef4d129" Nov 03 17:30:20.090742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.090711 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:20.096134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.096113 6001 scope.go:122] "RemoveContainer" containerID="22dd5918a13280e80402ba87437cea3a0dc1394736a52f426fb7a3675af34648" Nov 03 17:30:20.131934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.131892 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:20.159154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.159096 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" podStartSLOduration=20.15908041 podStartE2EDuration="20.15908041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:20.155904745 +0000 UTC m=+4515.489118315" watchObservedRunningTime="2026-11-03 17:30:20.15908041 +0000 UTC m=+4515.492293979" Nov 03 17:30:20.170949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.170917 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:20.215354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.215319 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:20.785123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.785085 6001 scope.go:122] "RemoveContainer" containerID="050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816" Nov 03 17:30:20.785832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.785805 6001 scope.go:122] "RemoveContainer" containerID="3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe" Nov 03 17:30:20.786189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.786158 6001 scope.go:122] "RemoveContainer" containerID="9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d" Nov 03 17:30:20.787180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.787034 6001 scope.go:122] "RemoveContainer" containerID="9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d" Nov 03 17:30:20.787527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.787507 6001 scope.go:122] "RemoveContainer" containerID="b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7" Nov 03 17:30:20.915535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.915507 6001 generic.go:356] "Generic (PLEG): container finished" podID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerID="beb07957d91a2506cd8b4e36d91f7aee98dbca8e7b82fc24d62201276efc37f9" exitCode=60 Nov 03 17:30:20.915934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.915573 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerDied","Data":"beb07957d91a2506cd8b4e36d91f7aee98dbca8e7b82fc24d62201276efc37f9"} Nov 03 17:30:20.915934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.915630 6001 scope.go:122] "RemoveContainer" containerID="c30c31745e42591059179b15e19d6bfb4a483350406e8922fdb87d5bae6966dc" Nov 03 17:30:20.916091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.916074 6001 scope.go:122] "RemoveContainer" containerID="beb07957d91a2506cd8b4e36d91f7aee98dbca8e7b82fc24d62201276efc37f9" Nov 03 17:30:20.916560 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:20.916530 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp_odh-ai-gateway-infra(e1547f64-57e7-4641-bee9-c1449c90b12f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" Nov 03 17:30:20.917944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.917923 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerStarted","Data":"245db185d569f9c54411390599ea0922b5b7f7c340216698eaf742f7e93b3f7d"} Nov 03 17:30:20.919963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.919940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerStarted","Data":"d33c8c3779f0f72cfd8dff98b5fa610947a8f17d9c26694a0d3a6b20f1e6575c"} Nov 03 17:30:20.926268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.926226 6001 generic.go:356] "Generic (PLEG): container finished" podID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerID="661efc795283e38c004e42860c4b8a65f941ee4e6f8df5a1fee37c5e05b6e597" exitCode=60 Nov 03 17:30:20.926385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.926367 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerDied","Data":"661efc795283e38c004e42860c4b8a65f941ee4e6f8df5a1fee37c5e05b6e597"} Nov 03 17:30:20.926804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.926784 6001 scope.go:122] "RemoveContainer" containerID="661efc795283e38c004e42860c4b8a65f941ee4e6f8df5a1fee37c5e05b6e597" Nov 03 17:30:20.927218 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:20.927192 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll_odh-ai-gateway-infra(b02a0e58-8cb5-4057-ac74-4b52d64f3176)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" Nov 03 17:30:20.940351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.936017 6001 scope.go:122] "RemoveContainer" containerID="06d3fde087ae3e67e4b71c8032e16e1637d812d10efce067be6372e7c957aab8" Nov 03 17:30:20.946401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:20.946326 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerStarted","Data":"71d94032fe98b35ec5a5d8876da304756afce1c4bfed6b00432680340efbc9f4"} Nov 03 17:30:21.265836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.265730 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:21.292506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.291498 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:21.358253 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.357806 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:21.365007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.364976 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:21.386685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.386655 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:21.431480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431422 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\" (UniqueName: \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\") pod \"47398737-8736-4f4b-b8d4-9ba4d716ea06\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " Nov 03 17:30:21.431687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431577 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\" (UniqueName: \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\") pod \"426d8922-d11b-4983-8b2d-52467d60dade\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " Nov 03 17:30:21.431687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431652 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\") pod \"46fac7a2-4eda-4e5c-be12-531425e2e807\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " Nov 03 17:30:21.431821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431804 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\") pod \"426d8922-d11b-4983-8b2d-52467d60dade\" (UID: \"426d8922-d11b-4983-8b2d-52467d60dade\") " Nov 03 17:30:21.431912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431872 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\") pod \"47398737-8736-4f4b-b8d4-9ba4d716ea06\" (UID: \"47398737-8736-4f4b-b8d4-9ba4d716ea06\") " Nov 03 17:30:21.431992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.431976 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\" (UniqueName: \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\") pod \"46fac7a2-4eda-4e5c-be12-531425e2e807\" (UID: \"46fac7a2-4eda-4e5c-be12-531425e2e807\") " Nov 03 17:30:21.433820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.433615 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca" pod "46fac7a2-4eda-4e5c-be12-531425e2e807" (UID: "46fac7a2-4eda-4e5c-be12-531425e2e807"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.433820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.433616 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca" pod "426d8922-d11b-4983-8b2d-52467d60dade" (UID: "426d8922-d11b-4983-8b2d-52467d60dade"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.434361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.434336 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca" pod "47398737-8736-4f4b-b8d4-9ba4d716ea06" (UID: "47398737-8736-4f4b-b8d4-9ba4d716ea06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.436290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.436260 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq" pod "46fac7a2-4eda-4e5c-be12-531425e2e807" (UID: "46fac7a2-4eda-4e5c-be12-531425e2e807"). InnerVolumeSpecName "kube-api-access-zz6dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.439101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.439065 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8" pod "426d8922-d11b-4983-8b2d-52467d60dade" (UID: "426d8922-d11b-4983-8b2d-52467d60dade"). InnerVolumeSpecName "kube-api-access-6z5w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.476997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.476956 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc" pod "47398737-8736-4f4b-b8d4-9ba4d716ea06" (UID: "47398737-8736-4f4b-b8d4-9ba4d716ea06"). InnerVolumeSpecName "kube-api-access-9wqvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.506569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.506540 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.533667 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\" (UniqueName: \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\") pod \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.533966 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\") pod \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\" (UID: \"1906ff6e-c536-416b-a1e6-c1995cc23c6a\") " Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534737 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz6dq\" (UniqueName: \"kubernetes.io/projected/46fac7a2-4eda-4e5c-be12-531425e2e807-kube-api-access-zz6dq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534762 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqvc\" (UniqueName: \"kubernetes.io/projected/47398737-8736-4f4b-b8d4-9ba4d716ea06-kube-api-access-9wqvc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534775 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z5w8\" (UniqueName: \"kubernetes.io/projected/426d8922-d11b-4983-8b2d-52467d60dade-kube-api-access-6z5w8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534789 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fac7a2-4eda-4e5c-be12-531425e2e807-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534801 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/426d8922-d11b-4983-8b2d-52467d60dade-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.534814 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47398737-8736-4f4b-b8d4-9ba4d716ea06-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.535532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.535159 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca" pod "1906ff6e-c536-416b-a1e6-c1995cc23c6a" (UID: "1906ff6e-c536-416b-a1e6-c1995cc23c6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.557227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.557194 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:21.557393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.557328 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8" pod "1906ff6e-c536-416b-a1e6-c1995cc23c6a" (UID: "1906ff6e-c536-416b-a1e6-c1995cc23c6a"). InnerVolumeSpecName "kube-api-access-lgxt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.559290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.559264 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:21.566086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.566015 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:21.571102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.571075 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:21.635790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.635751 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\") pod \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " Nov 03 17:30:21.636551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636523 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\") pod \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " Nov 03 17:30:21.636774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636635 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\" (UniqueName: \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\") pod \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\" (UID: \"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c\") " Nov 03 17:30:21.636774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636673 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\") pod \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " Nov 03 17:30:21.636774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636716 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\") pod \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636874 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\" (UniqueName: \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\") pod \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\" (UID: \"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2\") " Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636982 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\" (UniqueName: \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\") pod \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\" (UID: \"02617808-3d0b-4cbb-ac17-ec67fbac6ae1\") " Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.637014 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\" (UniqueName: \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\") pod \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\" (UID: \"dddbd6bf-5f2b-47a6-90c6-85daf77b6839\") " Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.636377 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca" pod "32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" (UID: "32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.638282 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca" pod "e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" (UID: "e4d7b1e1-09a5-4273-9cc0-2a278231ddf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.638409 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca" pod "02617808-3d0b-4cbb-ac17-ec67fbac6ae1" (UID: "02617808-3d0b-4cbb-ac17-ec67fbac6ae1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.638849 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca" pod "dddbd6bf-5f2b-47a6-90c6-85daf77b6839" (UID: "dddbd6bf-5f2b-47a6-90c6-85daf77b6839"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639343 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1906ff6e-c536-416b-a1e6-c1995cc23c6a-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639369 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639384 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639397 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxt8\" (UniqueName: \"kubernetes.io/projected/1906ff6e-c536-416b-a1e6-c1995cc23c6a-kube-api-access-lgxt8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639411 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.639424 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.642024 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5" pod "32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" (UID: "32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c"). InnerVolumeSpecName "kube-api-access-kbcq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.643084 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp" pod "dddbd6bf-5f2b-47a6-90c6-85daf77b6839" (UID: "dddbd6bf-5f2b-47a6-90c6-85daf77b6839"). InnerVolumeSpecName "kube-api-access-2rqzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.646891 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq" pod "02617808-3d0b-4cbb-ac17-ec67fbac6ae1" (UID: "02617808-3d0b-4cbb-ac17-ec67fbac6ae1"). InnerVolumeSpecName "kube-api-access-4ctsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.651702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.648915 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs" pod "e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" (UID: "e4d7b1e1-09a5-4273-9cc0-2a278231ddf2"). InnerVolumeSpecName "kube-api-access-rnfhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.740894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.740854 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbcq5\" (UniqueName: \"kubernetes.io/projected/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c-kube-api-access-kbcq5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.740894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.740897 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnfhs\" (UniqueName: \"kubernetes.io/projected/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2-kube-api-access-rnfhs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.741159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.740911 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ctsq\" (UniqueName: \"kubernetes.io/projected/02617808-3d0b-4cbb-ac17-ec67fbac6ae1-kube-api-access-4ctsq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.741159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.740924 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rqzp\" (UniqueName: \"kubernetes.io/projected/dddbd6bf-5f2b-47a6-90c6-85daf77b6839-kube-api-access-2rqzp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.967061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.967020 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerStarted","Data":"d25fb380dea7b8a94457e110e8a35781d57bd49042daf079e22a94f746290d72"} Nov 03 17:30:21.972925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.972887 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerStarted","Data":"10b5a2deb5fa2cfc9dd580e236879bf379cb9025af8cd95c92e7bc73efab600e"} Nov 03 17:30:21.975354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.975322 6001 generic.go:356] "Generic (PLEG): container finished" podID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerID="245db185d569f9c54411390599ea0922b5b7f7c340216698eaf742f7e93b3f7d" exitCode=60 Nov 03 17:30:21.975515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.975390 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerDied","Data":"245db185d569f9c54411390599ea0922b5b7f7c340216698eaf742f7e93b3f7d"} Nov 03 17:30:21.975515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.975496 6001 scope.go:122] "RemoveContainer" containerID="e355d14392351979e1bcf8f1fef2fa1d5639e80e05f3c6599b08f11116bd5b7f" Nov 03 17:30:21.975883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.975868 6001 scope.go:122] "RemoveContainer" containerID="245db185d569f9c54411390599ea0922b5b7f7c340216698eaf742f7e93b3f7d" Nov 03 17:30:21.976251 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:21.976231 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql_odh-ai-gateway-infra(a6a4ade1-ce40-48f5-8b65-94a72868c47a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" Nov 03 17:30:21.979060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.979031 6001 generic.go:356] "Generic (PLEG): container finished" podID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerID="d33c8c3779f0f72cfd8dff98b5fa610947a8f17d9c26694a0d3a6b20f1e6575c" exitCode=60 Nov 03 17:30:21.979208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.979107 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerDied","Data":"d33c8c3779f0f72cfd8dff98b5fa610947a8f17d9c26694a0d3a6b20f1e6575c"} Nov 03 17:30:21.979578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.979561 6001 scope.go:122] "RemoveContainer" containerID="d33c8c3779f0f72cfd8dff98b5fa610947a8f17d9c26694a0d3a6b20f1e6575c" Nov 03 17:30:21.979996 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:21.979975 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48_odh-ai-gateway-infra(0fdcae68-e0ee-4f64-9482-a5da2b888a0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" Nov 03 17:30:21.981009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.980992 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" Nov 03 17:30:21.981143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.980990 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25" event={"ID":"46fac7a2-4eda-4e5c-be12-531425e2e807","Type":"ContainerDied","Data":"30ca6428a83c0a69b7127d95ab94ea411e9eb6baf075e5b75ab147a09e768c6c"} Nov 03 17:30:21.982639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.982604 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" event={"ID":"426d8922-d11b-4983-8b2d-52467d60dade","Type":"ContainerDied","Data":"c63bcd26fa9411e1009b6432a1d27ff90e3a79c1146a55eb7b610968d7a3cfb1"} Nov 03 17:30:21.982778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.982637 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc" Nov 03 17:30:21.985084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.985061 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerStarted","Data":"90a371198536a55a965e2ec7d398ad66a0a15cf24b54bd44511d9a821b03764a"} Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.996611 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.996624 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr" event={"ID":"1906ff6e-c536-416b-a1e6-c1995cc23c6a","Type":"ContainerDied","Data":"4361c46c2489947154ed33d0ae104aef80886278b254d468e3326c44cae6fe80"} Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.997239 6001 scope.go:122] "RemoveContainer" containerID="24b93fb46b5255c36368b98f00e9101c88e43605f5211dec0aa754e21c71e558" Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.999691 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" event={"ID":"02617808-3d0b-4cbb-ac17-ec67fbac6ae1","Type":"ContainerDied","Data":"85100d9095f3e830c271e83cc672c3244716936678082de6daa74414c44ff092"} Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:21.999734 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc" Nov 03 17:30:22.003503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.002947 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" Nov 03 17:30:22.004286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.004265 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d" event={"ID":"dddbd6bf-5f2b-47a6-90c6-85daf77b6839","Type":"ContainerDied","Data":"0bd3d464d34fab2e6b36a282bb411e1ec98ca7fac6976a7b1291cd327fe023e5"} Nov 03 17:30:22.009988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.009687 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" event={"ID":"e4d7b1e1-09a5-4273-9cc0-2a278231ddf2","Type":"ContainerDied","Data":"bb73f17266719d10306b33b916e530e15fbb07b3f26f2a2e814fc06f14cde697"} Nov 03 17:30:22.013110 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.011060 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt" Nov 03 17:30:22.019604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.019581 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:22.024434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.023548 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerStarted","Data":"a9cc605d225291a011e72cbce7dd93879ca05ad819699db85f5710fd477644f6"} Nov 03 17:30:22.030084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.030051 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerStarted","Data":"9aaea41f8c8a52ac339ecaea4c2370f898d941988d6aca19ecfd60ec12b94ad7"} Nov 03 17:30:22.033345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.033158 6001 scope.go:122] "RemoveContainer" containerID="7d3374b39b25f206eaa917ca1a03e61951bd006eed868a64623beec09993ab6e" Nov 03 17:30:22.040922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.037663 6001 generic.go:356] "Generic (PLEG): container finished" podID="8565be49-1874-41cd-b659-aa5a09a24c74" containerID="71d94032fe98b35ec5a5d8876da304756afce1c4bfed6b00432680340efbc9f4" exitCode=60 Nov 03 17:30:22.040922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.037819 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerDied","Data":"71d94032fe98b35ec5a5d8876da304756afce1c4bfed6b00432680340efbc9f4"} Nov 03 17:30:22.040922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.038387 6001 scope.go:122] "RemoveContainer" containerID="71d94032fe98b35ec5a5d8876da304756afce1c4bfed6b00432680340efbc9f4" Nov 03 17:30:22.040922 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:22.038834 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh_odh-ai-gateway-infra(8565be49-1874-41cd-b659-aa5a09a24c74)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" Nov 03 17:30:22.045326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.044656 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" event={"ID":"32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c","Type":"ContainerDied","Data":"13b879b90ebec05d0b55be35093ecef959a53ddde831a089db297190f0e7ad64"} Nov 03 17:30:22.045326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.044799 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z" Nov 03 17:30:22.056667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.056641 6001 scope.go:122] "RemoveContainer" containerID="f3c36aa5b4097bf8ab633fe23d5c24237e5d10aafe724c23099ddf9fe86c4d84" Nov 03 17:30:22.060831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.060805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" event={"ID":"47398737-8736-4f4b-b8d4-9ba4d716ea06","Type":"ContainerDied","Data":"401fe9629c631484afde356919414a08ef5be4d1e8e621d2fce6b4b3c075c2fa"} Nov 03 17:30:22.061165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.061144 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq" Nov 03 17:30:22.109128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.107420 6001 scope.go:122] "RemoveContainer" containerID="153e62ae24b1a2f43923a3b63ab4f2ec48c4711c144db3ba083a8dd41f89fa58" Nov 03 17:30:22.157849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.157824 6001 scope.go:122] "RemoveContainer" containerID="18b681211a05ea2c8c3440dbae5b9526779ab9e146bd4dff7718cbc10ff696b0" Nov 03 17:30:22.198510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.198478 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:22.204478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.204299 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-460104-29895450-n2ggc"] Nov 03 17:30:22.261776 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:22.261694 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:30:22.283046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.283012 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:22.322058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.322019 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:22.326352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.326325 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895450-2qh25"] Nov 03 17:30:22.422603 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.422570 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:22.450787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.450767 6001 scope.go:122] "RemoveContainer" containerID="56b0ffb09f6911a6e4a66c106c3c0ace7264d9e0d61c0381d1c4b587329ab729" Nov 03 17:30:22.453850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.453827 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:22.458260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.458238 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:22.462669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.462650 6001 scope.go:122] "RemoveContainer" containerID="adeb0532abf4ebac66b2ae096dc8cc701121eca18df09b7ff51b56bb53ff622e" Nov 03 17:30:22.474368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.474345 6001 scope.go:122] "RemoveContainer" containerID="3e3fecabef1d0a02de470f9760aca18d782290f4db252d30509bc8ff633fe3c1" Nov 03 17:30:22.490005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.489983 6001 scope.go:122] "RemoveContainer" containerID="95a5ab97669799396f68506549dc12adbc43ac1e7f6db60c121d09037c6d38b7" Nov 03 17:30:22.501643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.501501 6001 scope.go:122] "RemoveContainer" containerID="e6fe56983038bf2035a2c90c96dca35213e5000042431a06125ae14810bb2eb2" Nov 03 17:30:22.505798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.505775 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:22.511428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.511398 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-2492c4-29895450-cdl6z"] Nov 03 17:30:22.542767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.542730 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:22.547180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.547145 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d6fcae-29895450-jlvwq"] Nov 03 17:30:22.567178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\") pod \"e1547f64-57e7-4641-bee9-c1449c90b12f\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " Nov 03 17:30:22.567356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567239 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\") pod \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " Nov 03 17:30:22.567356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567302 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\" (UniqueName: \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\") pod \"e1547f64-57e7-4641-bee9-c1449c90b12f\" (UID: \"e1547f64-57e7-4641-bee9-c1449c90b12f\") " Nov 03 17:30:22.567622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567599 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\" (UniqueName: \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\") pod \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\" (UID: \"b02a0e58-8cb5-4057-ac74-4b52d64f3176\") " Nov 03 17:30:22.567622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567608 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca" pod "e1547f64-57e7-4641-bee9-c1449c90b12f" (UID: "e1547f64-57e7-4641-bee9-c1449c90b12f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:22.567764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.567656 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca" pod "b02a0e58-8cb5-4057-ac74-4b52d64f3176" (UID: "b02a0e58-8cb5-4057-ac74-4b52d64f3176"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:22.568215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.568194 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1547f64-57e7-4641-bee9-c1449c90b12f-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:22.568215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.568216 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02a0e58-8cb5-4057-ac74-4b52d64f3176-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:22.569817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.569793 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx" pod "e1547f64-57e7-4641-bee9-c1449c90b12f" (UID: "e1547f64-57e7-4641-bee9-c1449c90b12f"). InnerVolumeSpecName "kube-api-access-wqwpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:22.569943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.569819 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv" pod "b02a0e58-8cb5-4057-ac74-4b52d64f3176" (UID: "b02a0e58-8cb5-4057-ac74-4b52d64f3176"). InnerVolumeSpecName "kube-api-access-jjgdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:22.584236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.584186 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:22.588480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.588429 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2b49df-29895450-h642d"] Nov 03 17:30:22.626565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.626532 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:22.632269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.632236 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-1e9d48-29895450-gkngr"] Nov 03 17:30:22.669740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.669662 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgdv\" (UniqueName: \"kubernetes.io/projected/b02a0e58-8cb5-4057-ac74-4b52d64f3176-kube-api-access-jjgdv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:22.669740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.669700 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwpx\" (UniqueName: \"kubernetes.io/projected/e1547f64-57e7-4641-bee9-c1449c90b12f-kube-api-access-wqwpx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:22.670340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.670315 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:22.674476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.674424 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-790111-29895450-fv4vc"] Nov 03 17:30:22.793286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.793251 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" path="/var/lib/kubelet/pods/02617808-3d0b-4cbb-ac17-ec67fbac6ae1/volumes" Nov 03 17:30:22.793809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.793792 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" path="/var/lib/kubelet/pods/1906ff6e-c536-416b-a1e6-c1995cc23c6a/volumes" Nov 03 17:30:22.794207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.794194 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" path="/var/lib/kubelet/pods/32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c/volumes" Nov 03 17:30:22.794976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.794960 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426d8922-d11b-4983-8b2d-52467d60dade" path="/var/lib/kubelet/pods/426d8922-d11b-4983-8b2d-52467d60dade/volumes" Nov 03 17:30:22.795369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.795355 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" path="/var/lib/kubelet/pods/46fac7a2-4eda-4e5c-be12-531425e2e807/volumes" Nov 03 17:30:22.795801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.795788 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" path="/var/lib/kubelet/pods/47398737-8736-4f4b-b8d4-9ba4d716ea06/volumes" Nov 03 17:30:22.796494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.796480 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" path="/var/lib/kubelet/pods/dddbd6bf-5f2b-47a6-90c6-85daf77b6839/volumes" Nov 03 17:30:22.816288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.815958 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:22.823351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:22.823321 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895450-8gzrt"] Nov 03 17:30:23.023209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.023179 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:23.071910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.071875 6001 generic.go:356] "Generic (PLEG): container finished" podID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerID="d25fb380dea7b8a94457e110e8a35781d57bd49042daf079e22a94f746290d72" exitCode=60 Nov 03 17:30:23.072105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.071959 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerDied","Data":"d25fb380dea7b8a94457e110e8a35781d57bd49042daf079e22a94f746290d72"} Nov 03 17:30:23.072105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.072026 6001 scope.go:122] "RemoveContainer" containerID="9dc85be2a56bef9beb9bf371d2202484d3265fa98ca5afb06119e199336fb21d" Nov 03 17:30:23.074228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.074202 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" event={"ID":"e1547f64-57e7-4641-bee9-c1449c90b12f","Type":"ContainerDied","Data":"7a24cc17ddf39af88a8ec20d1332e4e88bf4f22cc7a6021aabc88b6e105f2068"} Nov 03 17:30:23.074424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.074215 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp" Nov 03 17:30:23.076602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.076575 6001 generic.go:356] "Generic (PLEG): container finished" podID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerID="10b5a2deb5fa2cfc9dd580e236879bf379cb9025af8cd95c92e7bc73efab600e" exitCode=60 Nov 03 17:30:23.076759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.076655 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerDied","Data":"10b5a2deb5fa2cfc9dd580e236879bf379cb9025af8cd95c92e7bc73efab600e"} Nov 03 17:30:23.077170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.077151 6001 scope.go:122] "RemoveContainer" containerID="10b5a2deb5fa2cfc9dd580e236879bf379cb9025af8cd95c92e7bc73efab600e" Nov 03 17:30:23.077557 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:23.077536 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg_odh-ai-gateway-infra(d7d0bf64-0564-4905-8d4d-b932579d9f69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" Nov 03 17:30:23.086476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.086430 6001 generic.go:356] "Generic (PLEG): container finished" podID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerID="90a371198536a55a965e2ec7d398ad66a0a15cf24b54bd44511d9a821b03764a" exitCode=60 Nov 03 17:30:23.086642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.086530 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerDied","Data":"90a371198536a55a965e2ec7d398ad66a0a15cf24b54bd44511d9a821b03764a"} Nov 03 17:30:23.087027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.087003 6001 scope.go:122] "RemoveContainer" containerID="90a371198536a55a965e2ec7d398ad66a0a15cf24b54bd44511d9a821b03764a" Nov 03 17:30:23.087541 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:23.087509 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq_odh-ai-gateway-infra(cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" Nov 03 17:30:23.091161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.090400 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" Nov 03 17:30:23.091161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.090420 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll" event={"ID":"b02a0e58-8cb5-4057-ac74-4b52d64f3176","Type":"ContainerDied","Data":"b608ef700551f6bcde0bd287836f05dee43cfdb164bf671515b7a2340b0fa667"} Nov 03 17:30:23.091434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.091388 6001 scope.go:122] "RemoveContainer" containerID="beb07957d91a2506cd8b4e36d91f7aee98dbca8e7b82fc24d62201276efc37f9" Nov 03 17:30:23.099384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.099359 6001 generic.go:356] "Generic (PLEG): container finished" podID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerID="a9cc605d225291a011e72cbce7dd93879ca05ad819699db85f5710fd477644f6" exitCode=60 Nov 03 17:30:23.099492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.099423 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerDied","Data":"a9cc605d225291a011e72cbce7dd93879ca05ad819699db85f5710fd477644f6"} Nov 03 17:30:23.099928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.099908 6001 scope.go:122] "RemoveContainer" containerID="a9cc605d225291a011e72cbce7dd93879ca05ad819699db85f5710fd477644f6" Nov 03 17:30:23.100340 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:23.100314 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44_odh-ai-gateway-infra(c1c917b6-5e0b-4235-85a3-55acd85f2f6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" Nov 03 17:30:23.102483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.102458 6001 generic.go:356] "Generic (PLEG): container finished" podID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerID="9aaea41f8c8a52ac339ecaea4c2370f898d941988d6aca19ecfd60ec12b94ad7" exitCode=60 Nov 03 17:30:23.102587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.102537 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerDied","Data":"9aaea41f8c8a52ac339ecaea4c2370f898d941988d6aca19ecfd60ec12b94ad7"} Nov 03 17:30:23.102934 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.102920 6001 scope.go:122] "RemoveContainer" containerID="9aaea41f8c8a52ac339ecaea4c2370f898d941988d6aca19ecfd60ec12b94ad7" Nov 03 17:30:23.103230 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:23.103212 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r_odh-ai-gateway-infra(09e24410-4d9e-4116-93b5-aa72aac5672e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" Nov 03 17:30:23.116481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.115676 6001 scope.go:122] "RemoveContainer" containerID="b9248623bacea0be52ec0e3f3f4ba3086673160cbde6f5423b5b434fc214b7f7" Nov 03 17:30:23.140637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.140191 6001 scope.go:122] "RemoveContainer" containerID="050331fa2d1d19f048c112945b2a6584ad72a31ba3499998c3a20293f27bf816" Nov 03 17:30:23.185034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.184994 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:23.193936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.193910 6001 scope.go:122] "RemoveContainer" containerID="661efc795283e38c004e42860c4b8a65f941ee4e6f8df5a1fee37c5e05b6e597" Nov 03 17:30:23.207618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.207583 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:23.214940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.214872 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-da409b-29895450-qwhll"] Nov 03 17:30:23.221519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.221496 6001 scope.go:122] "RemoveContainer" containerID="9b68561e5c9ab330d344784bf732a05688e618eac80a36eb5a11c4653eec066d" Nov 03 17:30:23.254079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.254040 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:23.257211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.257183 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:23.263172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.263116 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5a5ab1-29895450-lzmmp"] Nov 03 17:30:23.264631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.264610 6001 scope.go:122] "RemoveContainer" containerID="3e1ff7779c89932e00baf071d9fc23b7580d18bdb2a175b332873a751ae3b7fe" Nov 03 17:30:23.328843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.328805 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:23.415609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.415577 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\" (UniqueName: \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\") pod \"3fa32e18-1a18-43ea-be00-01871c323b4c\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " Nov 03 17:30:23.415776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.415759 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\") pod \"3fa32e18-1a18-43ea-be00-01871c323b4c\" (UID: \"3fa32e18-1a18-43ea-be00-01871c323b4c\") " Nov 03 17:30:23.416108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.416083 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca" pod "3fa32e18-1a18-43ea-be00-01871c323b4c" (UID: "3fa32e18-1a18-43ea-be00-01871c323b4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:23.416312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.416295 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fa32e18-1a18-43ea-be00-01871c323b4c-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.417700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.417678 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl" pod "3fa32e18-1a18-43ea-be00-01871c323b4c" (UID: "3fa32e18-1a18-43ea-be00-01871c323b4c"). InnerVolumeSpecName "kube-api-access-774nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:23.422472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.422439 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:23.442229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.442205 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:23.446613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.446590 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:23.516781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.516750 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\") pod \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " Nov 03 17:30:23.517007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.516977 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\") pod \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " Nov 03 17:30:23.517104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.517018 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\") pod \"8565be49-1874-41cd-b659-aa5a09a24c74\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " Nov 03 17:30:23.517188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.517098 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\" (UniqueName: \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\") pod \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\" (UID: \"0fdcae68-e0ee-4f64-9482-a5da2b888a0a\") " Nov 03 17:30:23.517300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.517272 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\" (UniqueName: \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\") pod \"8565be49-1874-41cd-b659-aa5a09a24c74\" (UID: \"8565be49-1874-41cd-b659-aa5a09a24c74\") " Nov 03 17:30:23.517506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.517385 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\" (UniqueName: \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\") pod \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\" (UID: \"a6a4ade1-ce40-48f5-8b65-94a72868c47a\") " Nov 03 17:30:23.517668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.517644 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca" pod "0fdcae68-e0ee-4f64-9482-a5da2b888a0a" (UID: "0fdcae68-e0ee-4f64-9482-a5da2b888a0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:23.518226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.518168 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774nl\" (UniqueName: \"kubernetes.io/projected/3fa32e18-1a18-43ea-be00-01871c323b4c-kube-api-access-774nl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.518226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.518195 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.518846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.518819 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca" pod "8565be49-1874-41cd-b659-aa5a09a24c74" (UID: "8565be49-1874-41cd-b659-aa5a09a24c74"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:23.519828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.519806 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca" pod "a6a4ade1-ce40-48f5-8b65-94a72868c47a" (UID: "a6a4ade1-ce40-48f5-8b65-94a72868c47a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:23.520087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.520069 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w" pod "a6a4ade1-ce40-48f5-8b65-94a72868c47a" (UID: "a6a4ade1-ce40-48f5-8b65-94a72868c47a"). InnerVolumeSpecName "kube-api-access-rkt7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:23.520912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.520895 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8" pod "8565be49-1874-41cd-b659-aa5a09a24c74" (UID: "8565be49-1874-41cd-b659-aa5a09a24c74"). InnerVolumeSpecName "kube-api-access-ztbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:23.521362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.521346 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv" pod "0fdcae68-e0ee-4f64-9482-a5da2b888a0a" (UID: "0fdcae68-e0ee-4f64-9482-a5da2b888a0a"). InnerVolumeSpecName "kube-api-access-xvztv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:23.619736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.619639 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a4ade1-ce40-48f5-8b65-94a72868c47a-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.619736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.619678 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8565be49-1874-41cd-b659-aa5a09a24c74-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.619736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.619721 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvztv\" (UniqueName: \"kubernetes.io/projected/0fdcae68-e0ee-4f64-9482-a5da2b888a0a-kube-api-access-xvztv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.619736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.619733 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztbg8\" (UniqueName: \"kubernetes.io/projected/8565be49-1874-41cd-b659-aa5a09a24c74-kube-api-access-ztbg8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.620115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.619745 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkt7w\" (UniqueName: \"kubernetes.io/projected/a6a4ade1-ce40-48f5-8b65-94a72868c47a-kube-api-access-rkt7w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:23.723362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.723325 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:23.763712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:23.763678 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:24.137016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.136988 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" Nov 03 17:30:24.137371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.136985 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql" event={"ID":"a6a4ade1-ce40-48f5-8b65-94a72868c47a","Type":"ContainerDied","Data":"9d31451edf4143c58ea331d427beeac395459aa9d5e507065698c28c7b7f1570"} Nov 03 17:30:24.137371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.137162 6001 scope.go:122] "RemoveContainer" containerID="245db185d569f9c54411390599ea0922b5b7f7c340216698eaf742f7e93b3f7d" Nov 03 17:30:24.138573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.138547 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" Nov 03 17:30:24.138679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.138568 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48" event={"ID":"0fdcae68-e0ee-4f64-9482-a5da2b888a0a","Type":"ContainerDied","Data":"434bb0060558ac41efd9feff3a21a4b14c7aec9238cb2e47b2d365535a8881d9"} Nov 03 17:30:24.152693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.152667 6001 scope.go:122] "RemoveContainer" containerID="d33c8c3779f0f72cfd8dff98b5fa610947a8f17d9c26694a0d3a6b20f1e6575c" Nov 03 17:30:24.157037 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.156993 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" event={"ID":"8565be49-1874-41cd-b659-aa5a09a24c74","Type":"ContainerDied","Data":"1e9020a8a082102d76ebc0b284e0d3541408997dcfca44896a370c4ed939e6f2"} Nov 03 17:30:24.157162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.157134 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh" Nov 03 17:30:24.161790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.161596 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" Nov 03 17:30:24.161986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.161931 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4" event={"ID":"3fa32e18-1a18-43ea-be00-01871c323b4c","Type":"ContainerDied","Data":"fd7cd88db010bc4786fab47902ab7b38aeb6e13198cbd80b6e028fbc1d6a3b79"} Nov 03 17:30:24.171358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.171264 6001 scope.go:122] "RemoveContainer" containerID="71d94032fe98b35ec5a5d8876da304756afce1c4bfed6b00432680340efbc9f4" Nov 03 17:30:24.198381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.198250 6001 scope.go:122] "RemoveContainer" containerID="d25fb380dea7b8a94457e110e8a35781d57bd49042daf079e22a94f746290d72" Nov 03 17:30:24.284428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.284211 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:24.300329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.299933 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895450-znh48"] Nov 03 17:30:24.380507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.380477 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:24.385589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.385561 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:24.385901 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.385838 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895450-zhgql"] Nov 03 17:30:24.450599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.450567 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:24.456015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.455978 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:24.477926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.477899 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:24.497031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.496970 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-6046c2-29895450-68zqh"] Nov 03 17:30:24.529676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.529608 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:24.547508 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547476 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\") pod \"09e24410-4d9e-4116-93b5-aa72aac5672e\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " Nov 03 17:30:24.547726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547548 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\" (UniqueName: \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\") pod \"d7d0bf64-0564-4905-8d4d-b932579d9f69\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " Nov 03 17:30:24.547726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547640 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\") pod \"d7d0bf64-0564-4905-8d4d-b932579d9f69\" (UID: \"d7d0bf64-0564-4905-8d4d-b932579d9f69\") " Nov 03 17:30:24.547726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547688 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\" (UniqueName: \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\") pod \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " Nov 03 17:30:24.547905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547894 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\") pod \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\" (UID: \"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9\") " Nov 03 17:30:24.547974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.547926 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\") pod \"09e24410-4d9e-4116-93b5-aa72aac5672e\" (UID: \"09e24410-4d9e-4116-93b5-aa72aac5672e\") " Nov 03 17:30:24.548102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.548079 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca" pod "d7d0bf64-0564-4905-8d4d-b932579d9f69" (UID: "d7d0bf64-0564-4905-8d4d-b932579d9f69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:24.548395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.548367 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca" pod "cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" (UID: "cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:24.548622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.548401 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca" pod "09e24410-4d9e-4116-93b5-aa72aac5672e" (UID: "09e24410-4d9e-4116-93b5-aa72aac5672e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:24.548622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.548479 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d0bf64-0564-4905-8d4d-b932579d9f69-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.549941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.549912 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk" pod "d7d0bf64-0564-4905-8d4d-b932579d9f69" (UID: "d7d0bf64-0564-4905-8d4d-b932579d9f69"). InnerVolumeSpecName "kube-api-access-h6gwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:24.550488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.550424 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp" pod "cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" (UID: "cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9"). InnerVolumeSpecName "kube-api-access-j4pqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:24.550905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.550883 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:24.554659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.554632 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd067-29895450-d79j4"] Nov 03 17:30:24.556409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.556386 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2" pod "09e24410-4d9e-4116-93b5-aa72aac5672e" (UID: "09e24410-4d9e-4116-93b5-aa72aac5672e"). InnerVolumeSpecName "kube-api-access-2txx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:24.649568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.649489 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\") pod \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " Nov 03 17:30:24.649712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.649684 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\" (UniqueName: \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\") pod \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\" (UID: \"c1c917b6-5e0b-4235-85a3-55acd85f2f6f\") " Nov 03 17:30:24.649957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.649931 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca" pod "c1c917b6-5e0b-4235-85a3-55acd85f2f6f" (UID: "c1c917b6-5e0b-4235-85a3-55acd85f2f6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:24.650317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650295 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4pqp\" (UniqueName: \"kubernetes.io/projected/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-kube-api-access-j4pqp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.650414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650322 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.650414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650339 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.650414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650356 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09e24410-4d9e-4116-93b5-aa72aac5672e-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.650414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650370 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txx2\" (UniqueName: \"kubernetes.io/projected/09e24410-4d9e-4116-93b5-aa72aac5672e-kube-api-access-2txx2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.650414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.650385 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6gwk\" (UniqueName: \"kubernetes.io/projected/d7d0bf64-0564-4905-8d4d-b932579d9f69-kube-api-access-h6gwk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.653514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.653487 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m" pod "c1c917b6-5e0b-4235-85a3-55acd85f2f6f" (UID: "c1c917b6-5e0b-4235-85a3-55acd85f2f6f"). InnerVolumeSpecName "kube-api-access-hc26m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:24.751949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.751908 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc26m\" (UniqueName: \"kubernetes.io/projected/c1c917b6-5e0b-4235-85a3-55acd85f2f6f-kube-api-access-hc26m\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:24.795660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.795621 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" path="/var/lib/kubelet/pods/0fdcae68-e0ee-4f64-9482-a5da2b888a0a/volumes" Nov 03 17:30:24.796331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.796312 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" path="/var/lib/kubelet/pods/3fa32e18-1a18-43ea-be00-01871c323b4c/volumes" Nov 03 17:30:24.796791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.796775 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" path="/var/lib/kubelet/pods/8565be49-1874-41cd-b659-aa5a09a24c74/volumes" Nov 03 17:30:24.797468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.797454 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" path="/var/lib/kubelet/pods/a6a4ade1-ce40-48f5-8b65-94a72868c47a/volumes" Nov 03 17:30:24.797847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.797834 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" path="/var/lib/kubelet/pods/b02a0e58-8cb5-4057-ac74-4b52d64f3176/volumes" Nov 03 17:30:24.798213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.798202 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" path="/var/lib/kubelet/pods/e1547f64-57e7-4641-bee9-c1449c90b12f/volumes" Nov 03 17:30:24.798867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:24.798852 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" path="/var/lib/kubelet/pods/e4d7b1e1-09a5-4273-9cc0-2a278231ddf2/volumes" Nov 03 17:30:25.185319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.185278 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" event={"ID":"d7d0bf64-0564-4905-8d4d-b932579d9f69","Type":"ContainerDied","Data":"727b43df676128f112d5cb5b15a4c159749d4ef0d7c537121f2a99208f52f098"} Nov 03 17:30:25.185910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.185348 6001 scope.go:122] "RemoveContainer" containerID="10b5a2deb5fa2cfc9dd580e236879bf379cb9025af8cd95c92e7bc73efab600e" Nov 03 17:30:25.185910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.185288 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg" Nov 03 17:30:25.190172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.190144 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" Nov 03 17:30:25.190316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.190162 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq" event={"ID":"cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9","Type":"ContainerDied","Data":"8dc35b9b5261f507768042bb49399312ba641eabf75d6ccc8d1ad847796e91c8"} Nov 03 17:30:25.191965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.191942 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" Nov 03 17:30:25.192095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.191939 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44" event={"ID":"c1c917b6-5e0b-4235-85a3-55acd85f2f6f","Type":"ContainerDied","Data":"36a336b0472481fe2212680937b532cfdae1f8e6ee4ece01c02889c762fc0f0d"} Nov 03 17:30:25.194484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.193534 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" event={"ID":"09e24410-4d9e-4116-93b5-aa72aac5672e","Type":"ContainerDied","Data":"9ca62a4f2e51a8048176dbe55947ec4655542f70d5ace9384a90135626590f73"} Nov 03 17:30:25.194484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.193592 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r" Nov 03 17:30:25.200265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.200247 6001 scope.go:122] "RemoveContainer" containerID="90a371198536a55a965e2ec7d398ad66a0a15cf24b54bd44511d9a821b03764a" Nov 03 17:30:25.212714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.212686 6001 scope.go:122] "RemoveContainer" containerID="a9cc605d225291a011e72cbce7dd93879ca05ad819699db85f5710fd477644f6" Nov 03 17:30:25.226672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.226648 6001 scope.go:122] "RemoveContainer" containerID="9aaea41f8c8a52ac339ecaea4c2370f898d941988d6aca19ecfd60ec12b94ad7" Nov 03 17:30:25.274835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.274797 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:25.283021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.282987 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-71d7b4-29895450-nrt44"] Nov 03 17:30:25.320140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.320107 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:25.323759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.323734 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895450-mfm5r"] Nov 03 17:30:25.360365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.360328 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:25.364185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.364151 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-e2d835-29895450-q9hqg"] Nov 03 17:30:25.401992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.401953 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:25.406100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:25.406072 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-ca9d7c-29895450-k2jbq"] Nov 03 17:30:26.792575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:26.792531 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" path="/var/lib/kubelet/pods/09e24410-4d9e-4116-93b5-aa72aac5672e/volumes" Nov 03 17:30:26.793102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:26.793085 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" path="/var/lib/kubelet/pods/c1c917b6-5e0b-4235-85a3-55acd85f2f6f/volumes" Nov 03 17:30:26.793524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:26.793498 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" path="/var/lib/kubelet/pods/cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9/volumes" Nov 03 17:30:26.794227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:26.794212 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" path="/var/lib/kubelet/pods/d7d0bf64-0564-4905-8d4d-b932579d9f69/volumes" Nov 03 17:30:43.270338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:43.270300 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:30:48.271583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:48.271522 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" containerID="cri-o://c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22" gracePeriod=25 Nov 03 17:30:49.697765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:49.697721 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:30:49.730834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:49.730801 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:30:52.253031 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:52.252989 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:30:53.949283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:53.949245 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 17:30:53.985665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:53.985626 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 17:30:54.705503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:54.704887 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" podUID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" containerName="payload-processing" containerID="cri-o://7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad" gracePeriod=25 Nov 03 17:30:54.731705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:54.731653 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" podUID="e0e311c2-651e-47af-9fbd-4bcec31aad87" containerName="payload-pre-processing" containerID="cri-o://836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d" gracePeriod=25 Nov 03 17:30:55.030268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.030237 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:30:55.055660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.055636 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:30:55.072111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.072081 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\" (UniqueName: \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\") pod \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " Nov 03 17:30:55.072293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.072255 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\") pod \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\" (UID: \"bd3d69c2-fcb6-412d-9676-37cd88dd93de\") " Nov 03 17:30:55.073809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.073781 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume" pod "bd3d69c2-fcb6-412d-9676-37cd88dd93de" (UID: "bd3d69c2-fcb6-412d-9676-37cd88dd93de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:55.074080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.074061 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3d69c2-fcb6-412d-9676-37cd88dd93de-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:55.074876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.074853 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6" pod "bd3d69c2-fcb6-412d-9676-37cd88dd93de" (UID: "bd3d69c2-fcb6-412d-9676-37cd88dd93de"). InnerVolumeSpecName "kube-api-access-l2cs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:55.175512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.175424 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\" (UniqueName: \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\") pod \"e0e311c2-651e-47af-9fbd-4bcec31aad87\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " Nov 03 17:30:55.175688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.175608 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\") pod \"e0e311c2-651e-47af-9fbd-4bcec31aad87\" (UID: \"e0e311c2-651e-47af-9fbd-4bcec31aad87\") " Nov 03 17:30:55.176056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.176023 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume" pod "e0e311c2-651e-47af-9fbd-4bcec31aad87" (UID: "e0e311c2-651e-47af-9fbd-4bcec31aad87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:55.176056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.176035 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2cs6\" (UniqueName: \"kubernetes.io/projected/bd3d69c2-fcb6-412d-9676-37cd88dd93de-kube-api-access-l2cs6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:55.177557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.177531 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6" pod "e0e311c2-651e-47af-9fbd-4bcec31aad87" (UID: "e0e311c2-651e-47af-9fbd-4bcec31aad87"). InnerVolumeSpecName "kube-api-access-mbgw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:55.277756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.277675 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbgw6\" (UniqueName: \"kubernetes.io/projected/e0e311c2-651e-47af-9fbd-4bcec31aad87-kube-api-access-mbgw6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:55.277756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.277710 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0e311c2-651e-47af-9fbd-4bcec31aad87-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:30:55.568022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.567936 6001 generic.go:356] "Generic (PLEG): container finished" podID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" containerID="7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad" exitCode=0 Nov 03 17:30:55.568022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.567984 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" Nov 03 17:30:55.568299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.568027 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" event={"ID":"bd3d69c2-fcb6-412d-9676-37cd88dd93de","Type":"ContainerDied","Data":"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad"} Nov 03 17:30:55.568299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.568086 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k" event={"ID":"bd3d69c2-fcb6-412d-9676-37cd88dd93de","Type":"ContainerDied","Data":"93f8dabef487d0c77cb61f5df65d2e0e39319040f86e1f9d2b8f921d5aea3380"} Nov 03 17:30:55.568299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.568119 6001 scope.go:122] "RemoveContainer" containerID="7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad" Nov 03 17:30:55.569896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.569868 6001 generic.go:356] "Generic (PLEG): container finished" podID="e0e311c2-651e-47af-9fbd-4bcec31aad87" containerID="836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d" exitCode=0 Nov 03 17:30:55.570036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.569904 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" event={"ID":"e0e311c2-651e-47af-9fbd-4bcec31aad87","Type":"ContainerDied","Data":"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d"} Nov 03 17:30:55.570036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.569960 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" event={"ID":"e0e311c2-651e-47af-9fbd-4bcec31aad87","Type":"ContainerDied","Data":"d5eb042f8a83a55f1f7b46f176161db5a2e734490436fdc2c01379cda01feeed"} Nov 03 17:30:55.570036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.569911 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7" Nov 03 17:30:55.580298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.580281 6001 scope.go:122] "RemoveContainer" containerID="7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad" Nov 03 17:30:55.580596 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:55.580576 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad\": container with ID starting with 7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad not found: ID does not exist" containerID="7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad" Nov 03 17:30:55.580665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.580603 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad"} err="failed to get container status \"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad\": rpc error: code = NotFound desc = could not find container \"7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad\": container with ID starting with 7390862f4724d3736e5ea55bda0ffbfe76e0b886e73582a67121dce0bea594ad not found: ID does not exist" Nov 03 17:30:55.580665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.580620 6001 scope.go:122] "RemoveContainer" containerID="836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d" Nov 03 17:30:55.590963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.590942 6001 scope.go:122] "RemoveContainer" containerID="836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d" Nov 03 17:30:55.591230 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:30:55.591212 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d\": container with ID starting with 836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d not found: ID does not exist" containerID="836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d" Nov 03 17:30:55.591299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.591242 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d"} err="failed to get container status \"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d\": rpc error: code = NotFound desc = could not find container \"836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d\": container with ID starting with 836ab7aa3099956d6dd80ebea7957fbfe8703d981ebe7c657c671397a2ad543d not found: ID does not exist" Nov 03 17:30:55.622582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.622548 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:30:55.635162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.635129 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-06fa9cd7-68f4595f75-fnxr7"] Nov 03 17:30:55.661224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.661186 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:30:55.665143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.665117 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06fa9cd7-57fb6fd78c-l7d6k"] Nov 03 17:30:55.888845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.888759 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 17:30:55.889012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.888952 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" containerID="cri-o://6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782" gracePeriod=30 Nov 03 17:30:55.989148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:55.989110 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:56.790903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:56.790859 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" path="/var/lib/kubelet/pods/bd3d69c2-fcb6-412d-9676-37cd88dd93de/volumes" Nov 03 17:30:56.791585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:56.791561 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e311c2-651e-47af-9fbd-4bcec31aad87" path="/var/lib/kubelet/pods/e0e311c2-651e-47af-9fbd-4bcec31aad87/volumes" Nov 03 17:30:58.426028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:58.425992 6001 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:30:58.426474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:58.426051 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:30:58.950125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:58.950058 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" containerID="cri-o://0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae" gracePeriod=25 Nov 03 17:30:58.986719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:30:58.986659 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" containerID="cri-o://0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2" gracePeriod=25 Nov 03 17:31:01.253754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.253721 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 17:31:01.293499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293425 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293524 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293555 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293580 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293651 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293731 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.293967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293741 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.293967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293837 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.294135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293961 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.294135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.293973 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.294135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.294062 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.294135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.294129 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\") pod \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\" (UID: \"cc41d1b4-0f4d-45b7-bb96-cf584b04e721\") " Nov 03 17:31:01.294495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.294434 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.294733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.294705 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:01.295026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.295002 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.295119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.295033 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.295119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.295049 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.295119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.295063 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.295119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.295078 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.296117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.296087 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:01.296226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.296118 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:01.296226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.296126 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "kube-api-access-45mn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:01.296226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.296158 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy" pod "cc41d1b4-0f4d-45b7-bb96-cf584b04e721" (UID: "cc41d1b4-0f4d-45b7-bb96-cf584b04e721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:01.396512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.396468 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.396512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.396504 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.396512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.396516 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45mn9\" (UniqueName: \"kubernetes.io/projected/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-kube-api-access-45mn9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.396779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.396528 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc41d1b4-0f4d-45b7-bb96-cf584b04e721-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:01.665858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.665824 6001 generic.go:356] "Generic (PLEG): container finished" podID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerID="6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782" exitCode=0 Nov 03 17:31:01.666063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.665873 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" Nov 03 17:31:01.666063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.665903 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" event={"ID":"cc41d1b4-0f4d-45b7-bb96-cf584b04e721","Type":"ContainerDied","Data":"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782"} Nov 03 17:31:01.666063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.665964 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8" event={"ID":"cc41d1b4-0f4d-45b7-bb96-cf584b04e721","Type":"ContainerDied","Data":"57047bb39f659868f5485da79975d65dd55b96ab3d12cbe5505512ab92ed58fc"} Nov 03 17:31:01.666063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.665987 6001 scope.go:122] "RemoveContainer" containerID="6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782" Nov 03 17:31:01.679699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.679677 6001 scope.go:122] "RemoveContainer" containerID="6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782" Nov 03 17:31:01.680081 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:01.680045 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782\": container with ID starting with 6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782 not found: ID does not exist" containerID="6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782" Nov 03 17:31:01.680179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.680091 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782"} err="failed to get container status \"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782\": rpc error: code = NotFound desc = could not find container \"6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782\": container with ID starting with 6a05463c47481e24d6606e411f5fcad43a348cd600214e8a7541fb2b01f1a782 not found: ID does not exist" Nov 03 17:31:01.784203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.784169 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 17:31:01.839400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:01.839359 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-sw2l8"] Nov 03 17:31:02.791221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:02.791186 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" path="/var/lib/kubelet/pods/cc41d1b4-0f4d-45b7-bb96-cf584b04e721/volumes" Nov 03 17:31:04.055757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:04.055716 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 17:31:04.056464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:04.055969 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" containerID="cri-o://b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db" gracePeriod=30 Nov 03 17:31:05.049097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049065 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh"] Nov 03 17:31:05.049819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049789 6001 cpu_manager.go:414] "containerMap: removing container" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.049819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049815 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049835 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049841 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049856 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049864 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049877 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049886 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049895 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049901 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049911 6001 cpu_manager.go:414] "containerMap: removing container" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049917 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049938 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049945 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049958 6001 cpu_manager.go:414] "containerMap: removing container" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049964 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049977 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049984 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.049994 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050001 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050011 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050018 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050031 6001 cpu_manager.go:414] "containerMap: removing container" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.050034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050038 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050048 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050055 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050066 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050074 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050086 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050093 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050104 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050111 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050119 6001 cpu_manager.go:414] "containerMap: removing container" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050126 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050137 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050146 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050157 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050166 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050182 6001 cpu_manager.go:414] "containerMap: removing container" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050189 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050198 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050206 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050221 6001 cpu_manager.go:414] "containerMap: removing container" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050227 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050245 6001 cpu_manager.go:414] "containerMap: removing container" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050252 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050265 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050273 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050283 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050289 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050300 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050308 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050322 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050328 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050339 6001 cpu_manager.go:414] "containerMap: removing container" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050348 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050372 6001 cpu_manager.go:414] "containerMap: removing container" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050380 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050390 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050397 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050406 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050413 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050426 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050433 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050455 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050463 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050477 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050484 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050498 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050505 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050514 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050521 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050531 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050539 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050553 6001 cpu_manager.go:414] "containerMap: removing container" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050561 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050571 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050578 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050594 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050601 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050615 6001 cpu_manager.go:414] "containerMap: removing container" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050629 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050643 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e0e311c2-651e-47af-9fbd-4bcec31aad87" containerName="payload-pre-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050651 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e311c2-651e-47af-9fbd-4bcec31aad87" containerName="payload-pre-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050668 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050675 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050684 6001 cpu_manager.go:414] "containerMap: removing container" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050691 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050900 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050912 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050922 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050933 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050942 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050950 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050958 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050968 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050976 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e311c2-651e-47af-9fbd-4bcec31aad87" containerName="payload-pre-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050987 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.050995 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051003 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051013 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051022 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051034 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051042 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051051 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051062 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051070 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051079 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051090 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051098 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051107 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051117 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5684e128-aa30-430f-8604-8027740756cf" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051127 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051135 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051144 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc41d1b4-0f4d-45b7-bb96-cf584b04e721" containerName="istio-proxy" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051154 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051162 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051172 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051182 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051191 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051200 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051207 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051216 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051227 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051235 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3d69c2-fcb6-412d-9676-37cd88dd93de" containerName="payload-processing" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051243 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051253 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051265 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051275 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051282 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051290 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051298 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051307 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d0bf64-0564-4905-8d4d-b932579d9f69" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051316 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051325 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:05.051272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.051334 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7996d93d-b563-40ff-baeb-0a6e3f8ba227" containerName="payload-pre-processing" Nov 03 17:31:05.057137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.057118 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.076691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.076664 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh"] Nov 03 17:31:05.084186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.084163 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:31:05.084317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.084201 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:31:05.084317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.084263 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:31:05.085025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.085009 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:31:05.188845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.188811 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.189021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.188955 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.189119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.189024 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wknn\" (UniqueName: \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.290905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.290866 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.291091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.290946 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wknn\" (UniqueName: \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.291175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.291157 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.291617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.291596 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.305111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.305053 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:31:05.313883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.313853 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.314309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.314286 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wknn\" (UniqueName: \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\") pod \"maas-api-revoke-sub-ef080186-bqmrh\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.389911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.389880 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:31:05.396096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.396073 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:05.736096 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:31:05.736053 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a00b48f_d590_45e4_b9d3_5d17e48bb9cc.slice/crio-cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571 WatchSource:0}: Error finding container cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571: Status 404 returned error can't find the container with id cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571 Nov 03 17:31:05.736686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.736663 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh"] Nov 03 17:31:05.989360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:05.989269 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:06.717319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:06.717282 6001 generic.go:356] "Generic (PLEG): container finished" podID="0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" containerID="a23f33a7c105dfd468e2cae5c29114cf78bed136cc04039d0680fda5d48201f3" exitCode=0 Nov 03 17:31:06.717844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:06.717343 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" event={"ID":"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc","Type":"ContainerDied","Data":"a23f33a7c105dfd468e2cae5c29114cf78bed136cc04039d0680fda5d48201f3"} Nov 03 17:31:06.717844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:06.717381 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" event={"ID":"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc","Type":"ContainerStarted","Data":"cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571"} Nov 03 17:31:07.082467 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.082417 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:07.082667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.082468 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:07.907001 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.906974 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:07.942860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.942825 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\") pod \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " Nov 03 17:31:07.943093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.942893 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\") pod \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " Nov 03 17:31:07.943178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.943161 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\" (UniqueName: \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\") pod \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\" (UID: \"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc\") " Nov 03 17:31:07.943289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.943267 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca" pod "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" (UID: "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:07.943788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.943769 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:07.945103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.945066 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca" pod "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" (UID: "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:07.945340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:07.945324 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn" pod "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" (UID: "0a00b48f-d590-45e4-b9d3-5d17e48bb9cc"). InnerVolumeSpecName "kube-api-access-6wknn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:08.045105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.045016 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.045105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.045049 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wknn\" (UniqueName: \"kubernetes.io/projected/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc-kube-api-access-6wknn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.538793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.538765 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:31:08.557573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.557541 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\" (UniqueName: \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\") pod \"fdecf6a8-9484-43ea-86da-5757e5aabaab\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " Nov 03 17:31:08.557769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.557628 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\") pod \"fdecf6a8-9484-43ea-86da-5757e5aabaab\" (UID: \"fdecf6a8-9484-43ea-86da-5757e5aabaab\") " Nov 03 17:31:08.558053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.558015 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume" pod "fdecf6a8-9484-43ea-86da-5757e5aabaab" (UID: "fdecf6a8-9484-43ea-86da-5757e5aabaab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:08.558358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.558337 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdecf6a8-9484-43ea-86da-5757e5aabaab-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.559921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.559892 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv" pod "fdecf6a8-9484-43ea-86da-5757e5aabaab" (UID: "fdecf6a8-9484-43ea-86da-5757e5aabaab"). InnerVolumeSpecName "kube-api-access-trzgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:08.660187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.660093 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzgv\" (UniqueName: \"kubernetes.io/projected/fdecf6a8-9484-43ea-86da-5757e5aabaab-kube-api-access-trzgv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.735666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.735635 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" Nov 03 17:31:08.735666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.735642 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh" event={"ID":"0a00b48f-d590-45e4-b9d3-5d17e48bb9cc","Type":"ContainerDied","Data":"cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571"} Nov 03 17:31:08.735943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.735721 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb9e21479b5d29df03c36f20f4e1b4ba1e6a0d0c1b35e74b9cc2b4b357d21571" Nov 03 17:31:08.737278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.737253 6001 generic.go:356] "Generic (PLEG): container finished" podID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerID="c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22" exitCode=0 Nov 03 17:31:08.737418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.737289 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" Nov 03 17:31:08.737418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.737337 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" event={"ID":"fdecf6a8-9484-43ea-86da-5757e5aabaab","Type":"ContainerDied","Data":"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22"} Nov 03 17:31:08.737418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.737389 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr" event={"ID":"fdecf6a8-9484-43ea-86da-5757e5aabaab","Type":"ContainerDied","Data":"7040ba91445d809b760153e2828a526bdff6e2405018b8793dc619d6cf6ae99c"} Nov 03 17:31:08.737418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.737411 6001 scope.go:122] "RemoveContainer" containerID="c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22" Nov 03 17:31:08.749545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.749527 6001 scope.go:122] "RemoveContainer" containerID="c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22" Nov 03 17:31:08.749832 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:08.749811 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22\": container with ID starting with c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22 not found: ID does not exist" containerID="c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22" Nov 03 17:31:08.749880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.749840 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22"} err="failed to get container status \"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22\": rpc error: code = NotFound desc = could not find container \"c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22\": container with ID starting with c01f9cb860136e284bd0d4acfca58e565f7ac781712bcea696b0ec446c981d22 not found: ID does not exist" Nov 03 17:31:08.793666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.793636 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:31:08.796165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:08.796136 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6046c2-58979748dc-pmscr"] Nov 03 17:31:09.343142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.343070 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 17:31:09.378422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378388 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378528 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378595 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378631 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378694 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378735 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.378822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378736 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:09.378822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378792 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:09.379137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378854 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.379137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.378948 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.379137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379063 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\") pod \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\" (UID: \"86121e97-1bf0-4aaa-afc8-1474a7b6fd64\") " Nov 03 17:31:09.379137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379085 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:09.379137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379121 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:09.379418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379250 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:09.379782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379757 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.379782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379787 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.379958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379804 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.379958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379818 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.379958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.379833 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.381151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.381129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:09.381255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.381142 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:09.381255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.381146 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "kube-api-access-ht6hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:09.381378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.381257 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token" pod "86121e97-1bf0-4aaa-afc8-1474a7b6fd64" (UID: "86121e97-1bf0-4aaa-afc8-1474a7b6fd64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:09.481114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.481075 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.481114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.481108 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht6hh\" (UniqueName: \"kubernetes.io/projected/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-kube-api-access-ht6hh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.481114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.481120 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.481379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.481128 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86121e97-1bf0-4aaa-afc8-1474a7b6fd64-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:09.750105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.750068 6001 generic.go:356] "Generic (PLEG): container finished" podID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerID="b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db" exitCode=0 Nov 03 17:31:09.750301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.750120 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" Nov 03 17:31:09.750301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.750153 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerDied","Data":"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db"} Nov 03 17:31:09.750301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.750209 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx" event={"ID":"86121e97-1bf0-4aaa-afc8-1474a7b6fd64","Type":"ContainerDied","Data":"6b868484dc3ac2b9fd23a24360569d7cd00d1b24b7e1bb4dd5b6a0bdc2312b0a"} Nov 03 17:31:09.750301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.750234 6001 scope.go:122] "RemoveContainer" containerID="b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db" Nov 03 17:31:09.763162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.763139 6001 scope.go:122] "RemoveContainer" containerID="9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813" Nov 03 17:31:09.774772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.774753 6001 scope.go:122] "RemoveContainer" containerID="b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db" Nov 03 17:31:09.775088 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:09.775061 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db\": container with ID starting with b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db not found: ID does not exist" containerID="b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db" Nov 03 17:31:09.775145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.775090 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db"} err="failed to get container status \"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db\": rpc error: code = NotFound desc = could not find container \"b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db\": container with ID starting with b30e406cc5006fcbbb1f0b38a69248c8727e193dfe5f3b1fd5ffcbd7b6ae75db not found: ID does not exist" Nov 03 17:31:09.775145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.775109 6001 scope.go:122] "RemoveContainer" containerID="9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813" Nov 03 17:31:09.775404 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:09.775377 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813\": container with ID starting with 9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813 not found: ID does not exist" containerID="9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813" Nov 03 17:31:09.775488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.775404 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813"} err="failed to get container status \"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813\": rpc error: code = NotFound desc = could not find container \"9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813\": container with ID starting with 9cbb696f759123c42f7dcd00f3b8339ba0424309fbd0cc34d9df4f9c412f1813 not found: ID does not exist" Nov 03 17:31:09.806241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.806212 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 17:31:09.818551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:09.818525 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-001802-openshift-default-57dff5bc9-6h5wx"] Nov 03 17:31:10.039869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:10.039784 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84"] Nov 03 17:31:10.043164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:10.043141 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b8cbf0f-mjc84"] Nov 03 17:31:10.790819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:10.790785 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" path="/var/lib/kubelet/pods/86121e97-1bf0-4aaa-afc8-1474a7b6fd64/volumes" Nov 03 17:31:10.791504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:10.791461 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1190400-8d4a-4943-acd3-6d56fd59d03d" path="/var/lib/kubelet/pods/d1190400-8d4a-4943-acd3-6d56fd59d03d/volumes" Nov 03 17:31:10.791972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:10.791950 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" path="/var/lib/kubelet/pods/fdecf6a8-9484-43ea-86da-5757e5aabaab/volumes" Nov 03 17:31:13.396577 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:13.396544 6001 scope.go:122] "RemoveContainer" containerID="7a4f8315a97d996c7f13b8d3c17ae0b66a2ab28234d2df6f91e738a6854e8895" Nov 03 17:31:13.407723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:13.407700 6001 scope.go:122] "RemoveContainer" containerID="1dc3019421b168cc03a6b2fd0388e9ce659b5e67cfce306557093cfb6fb85c86" Nov 03 17:31:13.417298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:13.417279 6001 scope.go:122] "RemoveContainer" containerID="3432ae0d5a4c7829facfd1b7032cc5b037ff1eb244ce4440937031c20387a824" Nov 03 17:31:17.083384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:17.083342 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:17.083869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:17.083358 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:19.272889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.272845 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 17:31:19.316234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.316197 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 17:31:19.376807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.376772 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\") pod \"da6304a6-8964-4f43-b35a-0a98edbcf648\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " Nov 03 17:31:19.377009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.376823 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\" (UniqueName: \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\") pod \"da6304a6-8964-4f43-b35a-0a98edbcf648\" (UID: \"da6304a6-8964-4f43-b35a-0a98edbcf648\") " Nov 03 17:31:19.377274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.377244 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume" pod "da6304a6-8964-4f43-b35a-0a98edbcf648" (UID: "da6304a6-8964-4f43-b35a-0a98edbcf648"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:19.379116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.379090 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb" pod "da6304a6-8964-4f43-b35a-0a98edbcf648" (UID: "da6304a6-8964-4f43-b35a-0a98edbcf648"). InnerVolumeSpecName "kube-api-access-k46lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:19.478304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.478269 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\") pod \"f0db1692-459c-4b2c-8c20-440f391dadd7\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " Nov 03 17:31:19.478676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.478610 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\" (UniqueName: \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\") pod \"f0db1692-459c-4b2c-8c20-440f391dadd7\" (UID: \"f0db1692-459c-4b2c-8c20-440f391dadd7\") " Nov 03 17:31:19.478676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.478659 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume" pod "f0db1692-459c-4b2c-8c20-440f391dadd7" (UID: "f0db1692-459c-4b2c-8c20-440f391dadd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:19.479105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.479085 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0db1692-459c-4b2c-8c20-440f391dadd7-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.479176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.479111 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6304a6-8964-4f43-b35a-0a98edbcf648-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.479176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.479125 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46lb\" (UniqueName: \"kubernetes.io/projected/da6304a6-8964-4f43-b35a-0a98edbcf648-kube-api-access-k46lb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.481033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.481006 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp" pod "f0db1692-459c-4b2c-8c20-440f391dadd7" (UID: "f0db1692-459c-4b2c-8c20-440f391dadd7"). InnerVolumeSpecName "kube-api-access-9wlgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:19.580689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.580651 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wlgp\" (UniqueName: \"kubernetes.io/projected/f0db1692-459c-4b2c-8c20-440f391dadd7-kube-api-access-9wlgp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.874939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.874844 6001 generic.go:356] "Generic (PLEG): container finished" podID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerID="0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae" exitCode=0 Nov 03 17:31:19.874939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.874893 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" Nov 03 17:31:19.875215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.874928 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" event={"ID":"da6304a6-8964-4f43-b35a-0a98edbcf648","Type":"ContainerDied","Data":"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae"} Nov 03 17:31:19.875215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.874991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp" event={"ID":"da6304a6-8964-4f43-b35a-0a98edbcf648","Type":"ContainerDied","Data":"7287eae06a434f9bfd63a22d5ec75b2d2eeec10f096aea178caa126fc52b056c"} Nov 03 17:31:19.875215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.875021 6001 scope.go:122] "RemoveContainer" containerID="0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae" Nov 03 17:31:19.876730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.876702 6001 generic.go:356] "Generic (PLEG): container finished" podID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerID="0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2" exitCode=0 Nov 03 17:31:19.876887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.876770 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" Nov 03 17:31:19.876959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.876875 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" event={"ID":"f0db1692-459c-4b2c-8c20-440f391dadd7","Type":"ContainerDied","Data":"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2"} Nov 03 17:31:19.876959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.876940 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz" event={"ID":"f0db1692-459c-4b2c-8c20-440f391dadd7","Type":"ContainerDied","Data":"b22ce30c4c7bd6ba363a6c3c95983cc18d4ee7162cb5297263022e770ff9ab61"} Nov 03 17:31:19.887014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.886993 6001 scope.go:122] "RemoveContainer" containerID="0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae" Nov 03 17:31:19.887326 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:19.887294 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae\": container with ID starting with 0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae not found: ID does not exist" containerID="0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae" Nov 03 17:31:19.887430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.887323 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae"} err="failed to get container status \"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae\": rpc error: code = NotFound desc = could not find container \"0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae\": container with ID starting with 0b838b6c03358d9525795bc2d9a6b54ec2bf9c04cc80cca327fe3b672b11cbae not found: ID does not exist" Nov 03 17:31:19.887430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.887342 6001 scope.go:122] "RemoveContainer" containerID="0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2" Nov 03 17:31:19.899102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.899080 6001 scope.go:122] "RemoveContainer" containerID="0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2" Nov 03 17:31:19.899373 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:19.899355 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2\": container with ID starting with 0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2 not found: ID does not exist" containerID="0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2" Nov 03 17:31:19.899473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.899380 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2"} err="failed to get container status \"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2\": rpc error: code = NotFound desc = could not find container \"0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2\": container with ID starting with 0b7d70cddfd7d2a8b6f4d7be2e791a0a8e5748829c9378cc28c60addf1edd8b2 not found: ID does not exist" Nov 03 17:31:19.929385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.929343 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 17:31:19.941656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.941622 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-001802-54f68d8cd9-4zltp"] Nov 03 17:31:19.968716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.968685 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 17:31:19.972756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:19.972729 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-001802-6bfc787cdc-98jbz"] Nov 03 17:31:20.791274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:20.791238 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" path="/var/lib/kubelet/pods/da6304a6-8964-4f43-b35a-0a98edbcf648/volumes" Nov 03 17:31:20.791738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:20.791723 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" path="/var/lib/kubelet/pods/f0db1692-459c-4b2c-8c20-440f391dadd7/volumes" Nov 03 17:31:21.044115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:21.044022 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj"] Nov 03 17:31:21.047412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:21.047374 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0d7a7ab-2qvxj"] Nov 03 17:31:21.512629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:21.510686 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 17:31:21.549578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:21.549515 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 17:31:22.341037 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:22.340994 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:31:22.791459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.791414 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50662027-ba44-4166-9727-cc5ced001756" path="/var/lib/kubelet/pods/50662027-ba44-4166-9727-cc5ced001756/volumes" Nov 03 17:31:22.976829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.976801 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv"] Nov 03 17:31:22.977562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977539 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:22.977562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977556 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977570 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" containerName="revoke-keys" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977578 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" containerName="revoke-keys" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977594 6001 cpu_manager.go:414] "containerMap: removing container" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977600 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977609 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977614 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977626 6001 cpu_manager.go:414] "containerMap: removing container" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977631 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddbd6bf-5f2b-47a6-90c6-85daf77b6839" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977642 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977650 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977665 6001 cpu_manager.go:414] "containerMap: removing container" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977672 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977689 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977694 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c917b6-5e0b-4235-85a3-55acd85f2f6f" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977705 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977711 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a4ade1-ce40-48f5-8b65-94a72868c47a" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977719 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977724 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977733 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977738 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1547f64-57e7-4641-bee9-c1449c90b12f" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977754 6001 cpu_manager.go:414] "containerMap: removing container" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977759 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977774 6001 cpu_manager.go:414] "containerMap: removing container" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977779 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fac7a2-4eda-4e5c-be12-531425e2e807" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977794 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:22.977786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977799 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977807 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977811 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977817 6001 cpu_manager.go:414] "containerMap: removing container" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977824 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e24410-4d9e-4116-93b5-aa72aac5672e" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977838 6001 cpu_manager.go:414] "containerMap: removing container" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977843 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977853 6001 cpu_manager.go:414] "containerMap: removing container" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977857 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977862 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977867 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977876 6001 cpu_manager.go:414] "containerMap: removing container" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977880 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977894 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.977899 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978028 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cf3eae-23d4-4b97-bcf0-e6d6ebe4f63c" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978038 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff5ae3c-7c28-454a-8dfe-a1a6872cb1d9" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978044 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47398737-8736-4f4b-b8d4-9ba4d716ea06" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978054 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02a0e58-8cb5-4057-ac74-4b52d64f3176" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978065 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978075 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="da6304a6-8964-4f43-b35a-0a98edbcf648" containerName="payload-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978079 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0db1692-459c-4b2c-8c20-440f391dadd7" containerName="payload-pre-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978087 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fdcae68-e0ee-4f64-9482-a5da2b888a0a" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978092 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1906ff6e-c536-416b-a1e6-c1995cc23c6a" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978097 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d7b1e1-09a5-4273-9cc0-2a278231ddf2" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978104 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="426d8922-d11b-4983-8b2d-52467d60dade" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978111 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" containerName="revoke-keys" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978118 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="02617808-3d0b-4cbb-ac17-ec67fbac6ae1" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978125 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="86121e97-1bf0-4aaa-afc8-1474a7b6fd64" containerName="istio-proxy" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978134 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdecf6a8-9484-43ea-86da-5757e5aabaab" containerName="payload-pre-processing" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978142 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8565be49-1874-41cd-b659-aa5a09a24c74" containerName="cleanup" Nov 03 17:31:22.978710 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.978152 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa32e18-1a18-43ea-be00-01871c323b4c" containerName="cleanup" Nov 03 17:31:22.983747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:22.983725 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.002266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.002241 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv"] Nov 03 17:31:23.010819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.010795 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:31:23.011082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.011056 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:31:23.011196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.011127 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:31:23.013962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.013935 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:31:23.065584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.065421 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.065584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.065493 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.065867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.065636 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l28lx\" (UniqueName: \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.168332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.166822 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.168332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.166905 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.168332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.167106 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l28lx\" (UniqueName: \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.168728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.168665 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.203085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.203051 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.204936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.204910 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l28lx\" (UniqueName: \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\") pod \"maas-api-revoke-sub-a6cb243a-7p9sv\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.302274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.302241 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:23.650756 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:31:23.650723 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbd6dab_544b_4a88_91d5_8cb1e7018b15.slice/crio-0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5 WatchSource:0}: Error finding container 0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5: Status 404 returned error can't find the container with id 0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5 Nov 03 17:31:23.651226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.651168 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv"] Nov 03 17:31:23.951318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:23.951224 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" event={"ID":"cfbd6dab-544b-4a88-91d5-8cb1e7018b15","Type":"ContainerStarted","Data":"0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5"} Nov 03 17:31:24.961408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:24.961315 6001 generic.go:356] "Generic (PLEG): container finished" podID="cfbd6dab-544b-4a88-91d5-8cb1e7018b15" containerID="85f1b92a788718fedbe1942e9a07a8cf80fbc207169174dc9698772b38344ea0" exitCode=0 Nov 03 17:31:24.961408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:24.961382 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" event={"ID":"cfbd6dab-544b-4a88-91d5-8cb1e7018b15","Type":"ContainerDied","Data":"85f1b92a788718fedbe1942e9a07a8cf80fbc207169174dc9698772b38344ea0"} Nov 03 17:31:26.136337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.136308 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:26.221995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.221925 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\") pod \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " Nov 03 17:31:26.222176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.222098 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\") pod \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " Nov 03 17:31:26.222176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.222136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\" (UniqueName: \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\") pod \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\" (UID: \"cfbd6dab-544b-4a88-91d5-8cb1e7018b15\") " Nov 03 17:31:26.222600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.222555 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca" pod "cfbd6dab-544b-4a88-91d5-8cb1e7018b15" (UID: "cfbd6dab-544b-4a88-91d5-8cb1e7018b15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:26.222840 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.222816 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:26.224291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.224246 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca" pod "cfbd6dab-544b-4a88-91d5-8cb1e7018b15" (UID: "cfbd6dab-544b-4a88-91d5-8cb1e7018b15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:26.224918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.224891 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx" pod "cfbd6dab-544b-4a88-91d5-8cb1e7018b15" (UID: "cfbd6dab-544b-4a88-91d5-8cb1e7018b15"). InnerVolumeSpecName "kube-api-access-l28lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:26.324087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.324052 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:26.324087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.324082 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l28lx\" (UniqueName: \"kubernetes.io/projected/cfbd6dab-544b-4a88-91d5-8cb1e7018b15-kube-api-access-l28lx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:26.511432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.511353 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" containerID="cri-o://fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9" gracePeriod=25 Nov 03 17:31:26.549892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.549836 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" containerID="cri-o://65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc" gracePeriod=25 Nov 03 17:31:26.977945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.977908 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" event={"ID":"cfbd6dab-544b-4a88-91d5-8cb1e7018b15","Type":"ContainerDied","Data":"0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5"} Nov 03 17:31:26.977945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.977945 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv" Nov 03 17:31:26.978178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:26.977956 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd5103dc3e6958e7da083facb484c2c7c39b1fc761fdd69b2373bf95f3220c5" Nov 03 17:31:32.556607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:32.556562 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:33.571950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:33.571907 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:36.322514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:36.322481 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 17:31:36.356683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:36.356490 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 17:31:41.323022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:41.322852 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" containerID="cri-o://41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5" gracePeriod=25 Nov 03 17:31:41.356991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:41.356941 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" containerID="cri-o://3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0" gracePeriod=25 Nov 03 17:31:42.556545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:42.556508 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:43.571093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:43.571049 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:46.614228 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:46.614194 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a8048_eac8_4b9f_b676_f9bbd213bf0a.slice/crio-conmon-65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:31:46.837453 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.837415 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 17:31:46.888167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.888126 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 17:31:46.914993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.914957 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\") pod \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " Nov 03 17:31:46.915293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.915011 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\" (UniqueName: \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\") pod \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " Nov 03 17:31:46.915293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.915113 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\") pod \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\" (UID: \"6a578f35-4d6b-4672-b34e-cc18a37c7cea\") " Nov 03 17:31:46.915293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.915148 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\" (UniqueName: \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\") pod \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\" (UID: \"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a\") " Nov 03 17:31:46.915632 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.915598 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume" pod "ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" (UID: "ff4a8048-eac8-4b9f-b676-f9bbd213bf0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:46.915874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.915846 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume" pod "6a578f35-4d6b-4672-b34e-cc18a37c7cea" (UID: "6a578f35-4d6b-4672-b34e-cc18a37c7cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:46.916228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.916207 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:46.916334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.916234 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a578f35-4d6b-4672-b34e-cc18a37c7cea-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:46.918320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.918294 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq" pod "ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" (UID: "ff4a8048-eac8-4b9f-b676-f9bbd213bf0a"). InnerVolumeSpecName "kube-api-access-crfhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:46.918433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:46.918334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx" pod "6a578f35-4d6b-4672-b34e-cc18a37c7cea" (UID: "6a578f35-4d6b-4672-b34e-cc18a37c7cea"). InnerVolumeSpecName "kube-api-access-pvmqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:47.017761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.017711 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crfhq\" (UniqueName: \"kubernetes.io/projected/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a-kube-api-access-crfhq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:47.017761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.017752 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvmqx\" (UniqueName: \"kubernetes.io/projected/6a578f35-4d6b-4672-b34e-cc18a37c7cea-kube-api-access-pvmqx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:31:47.267704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.267667 6001 generic.go:356] "Generic (PLEG): container finished" podID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerID="65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc" exitCode=0 Nov 03 17:31:47.267938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.267895 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" event={"ID":"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a","Type":"ContainerDied","Data":"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc"} Nov 03 17:31:47.268036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.267959 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" event={"ID":"ff4a8048-eac8-4b9f-b676-f9bbd213bf0a","Type":"ContainerDied","Data":"068629b7a68afcbbea56d315c73a5bec3115fca8de83f2efa432626dcc2521b4"} Nov 03 17:31:47.268036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.267993 6001 scope.go:122] "RemoveContainer" containerID="65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc" Nov 03 17:31:47.268187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.268169 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6" Nov 03 17:31:47.281488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.276389 6001 generic.go:356] "Generic (PLEG): container finished" podID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerID="fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9" exitCode=0 Nov 03 17:31:47.281488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.277922 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" event={"ID":"6a578f35-4d6b-4672-b34e-cc18a37c7cea","Type":"ContainerDied","Data":"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9"} Nov 03 17:31:47.281488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.277989 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" event={"ID":"6a578f35-4d6b-4672-b34e-cc18a37c7cea","Type":"ContainerDied","Data":"eefdf6f69275e32b6497d6c9eeac1b4d6976fb0d8b1cc3b2d81f1cc13e440c6d"} Nov 03 17:31:47.281488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.278072 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g" Nov 03 17:31:47.315099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.315071 6001 scope.go:122] "RemoveContainer" containerID="65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc" Nov 03 17:31:47.315498 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:47.315463 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc\": container with ID starting with 65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc not found: ID does not exist" containerID="65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc" Nov 03 17:31:47.315561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.315508 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc"} err="failed to get container status \"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc\": rpc error: code = NotFound desc = could not find container \"65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc\": container with ID starting with 65fc20c738b62327aa0828ff361a95e0e1f771624b3c54c182eb46c450a17efc not found: ID does not exist" Nov 03 17:31:47.315561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.315529 6001 scope.go:122] "RemoveContainer" containerID="fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9" Nov 03 17:31:47.326498 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.326476 6001 scope.go:122] "RemoveContainer" containerID="fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9" Nov 03 17:31:47.326888 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:47.326866 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9\": container with ID starting with fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9 not found: ID does not exist" containerID="fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9" Nov 03 17:31:47.326939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.326897 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9"} err="failed to get container status \"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9\": rpc error: code = NotFound desc = could not find container \"fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9\": container with ID starting with fb18f8d3462d33564c59c7c4038ca8d91d35d540f77c2177ca538c1d3cc643a9 not found: ID does not exist" Nov 03 17:31:47.349361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.349329 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 17:31:47.352834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.352807 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7519b7-5b785f8dcc-6967g"] Nov 03 17:31:47.391754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.391714 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 17:31:47.401483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:47.401411 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-7519b7-5db5555547-mf7s6"] Nov 03 17:31:48.790546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:48.790508 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" path="/var/lib/kubelet/pods/6a578f35-4d6b-4672-b34e-cc18a37c7cea/volumes" Nov 03 17:31:48.791040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:48.791025 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" path="/var/lib/kubelet/pods/ff4a8048-eac8-4b9f-b676-f9bbd213bf0a/volumes" Nov 03 17:31:49.151611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:49.151517 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:49.151798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:49.151691 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:52.349515 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:31:52.349463 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:31:54.359538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:54.359502 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 17:31:54.392212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:54.392176 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 17:31:59.151477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:59.151414 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:59.151957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:59.151416 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:59.360582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:59.360543 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" containerID="cri-o://8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243" gracePeriod=25 Nov 03 17:31:59.393272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:31:59.393228 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" containerID="cri-o://bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3" gracePeriod=25 Nov 03 17:32:01.336810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.336774 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 17:32:01.337235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.337049 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" containerID="cri-o://e8b41ecbb320ce9cce3779c915dda4d7aa548a58cd711c7e6474df81186f7061" gracePeriod=30 Nov 03 17:32:01.495765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.495390 6001 patch_prober.go:29] interesting pod/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:32:01.495765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.495479 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:32:01.727797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.727764 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 17:32:01.731956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.731936 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 17:32:01.879683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.879584 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\") pod \"46d2b661-ff65-486a-aa75-5e27fc535040\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " Nov 03 17:32:01.879893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.879716 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\" (UniqueName: \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\") pod \"46d2b661-ff65-486a-aa75-5e27fc535040\" (UID: \"46d2b661-ff65-486a-aa75-5e27fc535040\") " Nov 03 17:32:01.879973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.879955 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\") pod \"6274d78b-7ebd-493e-9712-904c7320f5de\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " Nov 03 17:32:01.880078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.880039 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume" pod "46d2b661-ff65-486a-aa75-5e27fc535040" (UID: "46d2b661-ff65-486a-aa75-5e27fc535040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:01.880078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.880047 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\" (UniqueName: \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\") pod \"6274d78b-7ebd-493e-9712-904c7320f5de\" (UID: \"6274d78b-7ebd-493e-9712-904c7320f5de\") " Nov 03 17:32:01.880438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.880408 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume" pod "6274d78b-7ebd-493e-9712-904c7320f5de" (UID: "6274d78b-7ebd-493e-9712-904c7320f5de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:01.880792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.880774 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46d2b661-ff65-486a-aa75-5e27fc535040-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:01.880854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.880795 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6274d78b-7ebd-493e-9712-904c7320f5de-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:01.882341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.882321 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl" pod "6274d78b-7ebd-493e-9712-904c7320f5de" (UID: "6274d78b-7ebd-493e-9712-904c7320f5de"). InnerVolumeSpecName "kube-api-access-mrjxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:01.882425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.882349 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt" pod "46d2b661-ff65-486a-aa75-5e27fc535040" (UID: "46d2b661-ff65-486a-aa75-5e27fc535040"). InnerVolumeSpecName "kube-api-access-4nhgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:01.981980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.981946 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrjxl\" (UniqueName: \"kubernetes.io/projected/6274d78b-7ebd-493e-9712-904c7320f5de-kube-api-access-mrjxl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:01.981980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:01.981974 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nhgt\" (UniqueName: \"kubernetes.io/projected/46d2b661-ff65-486a-aa75-5e27fc535040-kube-api-access-4nhgt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.202939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.202861 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8"] Nov 03 17:32:02.203541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203526 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" Nov 03 17:32:02.203590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203542 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" Nov 03 17:32:02.203590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203573 6001 cpu_manager.go:414] "containerMap: removing container" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" Nov 03 17:32:02.203590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203578 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" Nov 03 17:32:02.203590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203590 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" Nov 03 17:32:02.203731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203596 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" Nov 03 17:32:02.203731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203630 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cfbd6dab-544b-4a88-91d5-8cb1e7018b15" containerName="revoke-keys" Nov 03 17:32:02.203731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203636 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfbd6dab-544b-4a88-91d5-8cb1e7018b15" containerName="revoke-keys" Nov 03 17:32:02.203731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203645 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" Nov 03 17:32:02.203731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203650 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" Nov 03 17:32:02.203913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203747 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" containerName="payload-processing" Nov 03 17:32:02.203913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203756 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a8048-eac8-4b9f-b676-f9bbd213bf0a" containerName="payload-pre-processing" Nov 03 17:32:02.203913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203770 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfbd6dab-544b-4a88-91d5-8cb1e7018b15" containerName="revoke-keys" Nov 03 17:32:02.203913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203775 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a578f35-4d6b-4672-b34e-cc18a37c7cea" containerName="payload-processing" Nov 03 17:32:02.203913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.203782 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" containerName="payload-pre-processing" Nov 03 17:32:02.207553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.207532 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.224061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.224037 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:32:02.224248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.224168 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:32:02.229288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.229264 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8"] Nov 03 17:32:02.304268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.304235 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:32:02.304268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.304268 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:32:02.388186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.388148 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.388612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.388347 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bnd\" (UniqueName: \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.388612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.388408 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.490345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.490314 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.490572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.490439 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.490654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.490637 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bnd\" (UniqueName: \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.491154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.491134 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.492843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.492820 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.494173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.494145 6001 generic.go:356] "Generic (PLEG): container finished" podID="46d2b661-ff65-486a-aa75-5e27fc535040" containerID="41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5" exitCode=0 Nov 03 17:32:02.494311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.494198 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" Nov 03 17:32:02.494311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.494239 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" event={"ID":"46d2b661-ff65-486a-aa75-5e27fc535040","Type":"ContainerDied","Data":"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5"} Nov 03 17:32:02.494311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.494300 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh" event={"ID":"46d2b661-ff65-486a-aa75-5e27fc535040","Type":"ContainerDied","Data":"d5cdc147e310eb7e6a6dbb15680b34ab17fffd96bf32547eac2f4d32cca348cd"} Nov 03 17:32:02.494529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.494331 6001 scope.go:122] "RemoveContainer" containerID="41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5" Nov 03 17:32:02.496017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.495989 6001 generic.go:356] "Generic (PLEG): container finished" podID="6274d78b-7ebd-493e-9712-904c7320f5de" containerID="3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0" exitCode=0 Nov 03 17:32:02.496133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.496032 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" event={"ID":"6274d78b-7ebd-493e-9712-904c7320f5de","Type":"ContainerDied","Data":"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0"} Nov 03 17:32:02.496133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.496045 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" Nov 03 17:32:02.496133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.496071 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt" event={"ID":"6274d78b-7ebd-493e-9712-904c7320f5de","Type":"ContainerDied","Data":"826579c738753597d25cd4a1794c83eaa584c015988fd3ee208608df815b39fe"} Nov 03 17:32:02.513041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.513024 6001 scope.go:122] "RemoveContainer" containerID="41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5" Nov 03 17:32:02.513321 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:02.513304 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5\": container with ID starting with 41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5 not found: ID does not exist" containerID="41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5" Nov 03 17:32:02.513395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.513329 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5"} err="failed to get container status \"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5\": rpc error: code = NotFound desc = could not find container \"41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5\": container with ID starting with 41b56e7343bbd81b3894632db2c838408508e5fd3073876a2252ac82f0605fa5 not found: ID does not exist" Nov 03 17:32:02.513395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.513345 6001 scope.go:122] "RemoveContainer" containerID="3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0" Nov 03 17:32:02.514326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.514314 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bnd\" (UniqueName: \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\") pod \"maas-api-revoke-sub-7cf208a7-d72p8\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.521590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.521572 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:02.535628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.535611 6001 scope.go:122] "RemoveContainer" containerID="3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0" Nov 03 17:32:02.535882 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:02.535863 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0\": container with ID starting with 3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0 not found: ID does not exist" containerID="3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0" Nov 03 17:32:02.535948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.535890 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0"} err="failed to get container status \"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0\": rpc error: code = NotFound desc = could not find container \"3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0\": container with ID starting with 3bce4959961f399d832df9668f0ce69b00ea896dbb0afaf675d791035bbffda0 not found: ID does not exist" Nov 03 17:32:02.580131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.577651 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 17:32:02.584111 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.584078 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd067-6c445d5fcf-dtktt"] Nov 03 17:32:02.609313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.609266 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 17:32:02.616482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.613775 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd067-74d68576c-mlcvh"] Nov 03 17:32:02.689951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.689912 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:02.790587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.790508 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d2b661-ff65-486a-aa75-5e27fc535040" path="/var/lib/kubelet/pods/46d2b661-ff65-486a-aa75-5e27fc535040/volumes" Nov 03 17:32:02.790993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.790979 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6274d78b-7ebd-493e-9712-904c7320f5de" path="/var/lib/kubelet/pods/6274d78b-7ebd-493e-9712-904c7320f5de/volumes" Nov 03 17:32:02.843206 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:32:02.843171 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068415a3_cd0f_4b74_9c9c_6a1684da815d.slice/crio-70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518 WatchSource:0}: Error finding container 70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518: Status 404 returned error can't find the container with id 70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518 Nov 03 17:32:02.843206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:02.843193 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8"] Nov 03 17:32:03.535609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:03.535566 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" event={"ID":"068415a3-cd0f-4b74-9c9c-6a1684da815d","Type":"ContainerStarted","Data":"5159bd2c33115818785d90d59b78127385990a3567962ea3f216d811b1937981"} Nov 03 17:32:03.536090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:03.535636 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" event={"ID":"068415a3-cd0f-4b74-9c9c-6a1684da815d","Type":"ContainerStarted","Data":"70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518"} Nov 03 17:32:03.579378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:03.579322 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" podStartSLOduration=1.579307418 podStartE2EDuration="1.579307418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:03.576950725 +0000 UTC m=+4618.910164317" watchObservedRunningTime="2026-11-03 17:32:03.579307418 +0000 UTC m=+4618.912521003" Nov 03 17:32:03.697404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:03.697369 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:04.544617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:04.544580 6001 generic.go:356] "Generic (PLEG): container finished" podID="068415a3-cd0f-4b74-9c9c-6a1684da815d" containerID="5159bd2c33115818785d90d59b78127385990a3567962ea3f216d811b1937981" exitCode=0 Nov 03 17:32:04.545092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:04.544669 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" event={"ID":"068415a3-cd0f-4b74-9c9c-6a1684da815d","Type":"ContainerDied","Data":"5159bd2c33115818785d90d59b78127385990a3567962ea3f216d811b1937981"} Nov 03 17:32:05.721186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.721156 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:05.746504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.746467 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\") pod \"068415a3-cd0f-4b74-9c9c-6a1684da815d\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " Nov 03 17:32:05.746728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.746549 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\" (UniqueName: \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\") pod \"068415a3-cd0f-4b74-9c9c-6a1684da815d\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " Nov 03 17:32:05.746804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.746758 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\") pod \"068415a3-cd0f-4b74-9c9c-6a1684da815d\" (UID: \"068415a3-cd0f-4b74-9c9c-6a1684da815d\") " Nov 03 17:32:05.746867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.746797 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca" pod "068415a3-cd0f-4b74-9c9c-6a1684da815d" (UID: "068415a3-cd0f-4b74-9c9c-6a1684da815d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:05.747163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.747144 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:05.748764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.748739 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd" pod "068415a3-cd0f-4b74-9c9c-6a1684da815d" (UID: "068415a3-cd0f-4b74-9c9c-6a1684da815d"). InnerVolumeSpecName "kube-api-access-65bnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:05.748858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.748769 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca" pod "068415a3-cd0f-4b74-9c9c-6a1684da815d" (UID: "068415a3-cd0f-4b74-9c9c-6a1684da815d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:05.849203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.849104 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068415a3-cd0f-4b74-9c9c-6a1684da815d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:05.849203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:05.849142 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65bnd\" (UniqueName: \"kubernetes.io/projected/068415a3-cd0f-4b74-9c9c-6a1684da815d-kube-api-access-65bnd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.564348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.564309 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" event={"ID":"068415a3-cd0f-4b74-9c9c-6a1684da815d","Type":"ContainerDied","Data":"70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518"} Nov 03 17:32:06.564582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.564378 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70892ef0055261392451dcb64fd6bac0900baf65c6e729086e8403eaca9d6518" Nov 03 17:32:06.564582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.564324 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8" Nov 03 17:32:06.569637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.569309 6001 generic.go:356] "Generic (PLEG): container finished" podID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerID="e8b41ecbb320ce9cce3779c915dda4d7aa548a58cd711c7e6474df81186f7061" exitCode=0 Nov 03 17:32:06.569637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.569438 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerDied","Data":"e8b41ecbb320ce9cce3779c915dda4d7aa548a58cd711c7e6474df81186f7061"} Nov 03 17:32:06.569637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.569530 6001 scope.go:122] "RemoveContainer" containerID="e75a3635e62f272c3399906098a6c63712260b05274e4d24c51d5264fd6bb34a" Nov 03 17:32:06.603194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.603128 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 17:32:06.661522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.661721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661609 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.661784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661734 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.661784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661771 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.661899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661829 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.661958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661910 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.662011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.661964 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.662065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.662028 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.662131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.662112 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\") pod \"4e5abaaf-cd21-4be2-9435-53e2180a8229\" (UID: \"4e5abaaf-cd21-4be2-9435-53e2180a8229\") " Nov 03 17:32:06.665892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.665856 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:06.666400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.666374 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:06.668171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.667778 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:06.668171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.668004 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:06.670231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.668780 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:06.670682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.670649 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:32:06.672571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.672541 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:06.674691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.674662 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:06.674691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.674658 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr" pod "4e5abaaf-cd21-4be2-9435-53e2180a8229" (UID: "4e5abaaf-cd21-4be2-9435-53e2180a8229"). InnerVolumeSpecName "kube-api-access-w6rwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:06.764154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764102 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764151 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rwr\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-kube-api-access-w6rwr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764162 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5abaaf-cd21-4be2-9435-53e2180a8229-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764172 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764182 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764191 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764199 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764211 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.764686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.764223 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5abaaf-cd21-4be2-9435-53e2180a8229-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:06.919151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.919062 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 17:32:06.927204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.927164 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 17:32:06.947542 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:06.947505 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 17:32:07.578527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:07.578490 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" Nov 03 17:32:07.578527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:07.578496 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x" event={"ID":"4e5abaaf-cd21-4be2-9435-53e2180a8229","Type":"ContainerDied","Data":"a14397b0e15ab0eecac3fa57c547184d7d0f0c49e30278c38cf6cf3bec5950b7"} Nov 03 17:32:07.578767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:07.578556 6001 scope.go:122] "RemoveContainer" containerID="e8b41ecbb320ce9cce3779c915dda4d7aa548a58cd711c7e6474df81186f7061" Nov 03 17:32:07.633077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:07.633037 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 17:32:07.645362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:07.645327 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-2b49df-openshift-default-6756c66c6f-2t99x"] Nov 03 17:32:08.790258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:08.790220 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" path="/var/lib/kubelet/pods/4e5abaaf-cd21-4be2-9435-53e2180a8229/volumes" Nov 03 17:32:12.688815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:12.688767 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:13.588666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:13.588630 6001 scope.go:122] "RemoveContainer" containerID="7d48b44b3630f2d98d087609dd5b8cd10144975f9c5ad6492b085b775cfa1cfd" Nov 03 17:32:13.599576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:13.599554 6001 scope.go:122] "RemoveContainer" containerID="61d2b7c6029fa59652314ea23927157b01f7a75f88a22d4e6542ccf9973e8c05" Nov 03 17:32:13.697627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:13.697584 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:19.699075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.699046 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 17:32:19.722851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.722824 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 17:32:19.727879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.727849 6001 generic.go:356] "Generic (PLEG): container finished" podID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerID="bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3" exitCode=0 Nov 03 17:32:19.727989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.727890 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" Nov 03 17:32:19.727989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.727932 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" event={"ID":"8e0014c4-3df4-4657-8350-9db0c4ad78c6","Type":"ContainerDied","Data":"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3"} Nov 03 17:32:19.727989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.727982 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg" event={"ID":"8e0014c4-3df4-4657-8350-9db0c4ad78c6","Type":"ContainerDied","Data":"1539b1048b86d202d658232810f6c1d912a973d447931b2266488c1a8f776516"} Nov 03 17:32:19.728146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.728005 6001 scope.go:122] "RemoveContainer" containerID="bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3" Nov 03 17:32:19.729568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.729549 6001 generic.go:356] "Generic (PLEG): container finished" podID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerID="8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243" exitCode=0 Nov 03 17:32:19.729672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.729592 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" Nov 03 17:32:19.729672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.729633 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" event={"ID":"a6bdf97b-8183-4878-ac24-9c88fd170cdb","Type":"ContainerDied","Data":"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243"} Nov 03 17:32:19.729760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.729678 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h" event={"ID":"a6bdf97b-8183-4878-ac24-9c88fd170cdb","Type":"ContainerDied","Data":"7459834ba0e09aedd045b12e6847d6c9860b00a02de521a7653950b231b8e68a"} Nov 03 17:32:19.738389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.738371 6001 scope.go:122] "RemoveContainer" containerID="bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3" Nov 03 17:32:19.738704 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:19.738685 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3\": container with ID starting with bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3 not found: ID does not exist" containerID="bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3" Nov 03 17:32:19.738757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.738711 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3"} err="failed to get container status \"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3\": rpc error: code = NotFound desc = could not find container \"bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3\": container with ID starting with bcc5c206bbb80f99fd88aed3a7822d2e7268054b06c23e0223b3baca3da500b3 not found: ID does not exist" Nov 03 17:32:19.738757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.738728 6001 scope.go:122] "RemoveContainer" containerID="8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243" Nov 03 17:32:19.752070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.752045 6001 scope.go:122] "RemoveContainer" containerID="8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243" Nov 03 17:32:19.752571 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:19.752421 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243\": container with ID starting with 8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243 not found: ID does not exist" containerID="8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243" Nov 03 17:32:19.752571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.752466 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243"} err="failed to get container status \"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243\": rpc error: code = NotFound desc = could not find container \"8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243\": container with ID starting with 8c5c39fd105bc896f0df18dfa1291f1be343a35fc83429091f4d73f0d47ff243 not found: ID does not exist" Nov 03 17:32:19.858641 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.858604 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\" (UniqueName: \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\") pod \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " Nov 03 17:32:19.858798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.858674 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\") pod \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " Nov 03 17:32:19.858798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.858732 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\") pod \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\" (UID: \"8e0014c4-3df4-4657-8350-9db0c4ad78c6\") " Nov 03 17:32:19.858907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.858811 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\" (UniqueName: \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\") pod \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\" (UID: \"a6bdf97b-8183-4878-ac24-9c88fd170cdb\") " Nov 03 17:32:19.859081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.859056 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume" pod "a6bdf97b-8183-4878-ac24-9c88fd170cdb" (UID: "a6bdf97b-8183-4878-ac24-9c88fd170cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:19.859169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.859130 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume" pod "8e0014c4-3df4-4657-8350-9db0c4ad78c6" (UID: "8e0014c4-3df4-4657-8350-9db0c4ad78c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:19.859404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.859378 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6bdf97b-8183-4878-ac24-9c88fd170cdb-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:19.859404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.859400 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e0014c4-3df4-4657-8350-9db0c4ad78c6-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:19.860797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.860779 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm" pod "8e0014c4-3df4-4657-8350-9db0c4ad78c6" (UID: "8e0014c4-3df4-4657-8350-9db0c4ad78c6"). InnerVolumeSpecName "kube-api-access-bbscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:19.860892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.860867 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw" pod "a6bdf97b-8183-4878-ac24-9c88fd170cdb" (UID: "a6bdf97b-8183-4878-ac24-9c88fd170cdb"). InnerVolumeSpecName "kube-api-access-swzrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:19.961271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.961185 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbscm\" (UniqueName: \"kubernetes.io/projected/8e0014c4-3df4-4657-8350-9db0c4ad78c6-kube-api-access-bbscm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:19.961271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:19.961212 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzrw\" (UniqueName: \"kubernetes.io/projected/a6bdf97b-8183-4878-ac24-9c88fd170cdb-kube-api-access-swzrw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:20.032912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.032872 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw"] Nov 03 17:32:20.038060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.038032 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5110bcd2-w8wvw"] Nov 03 17:32:20.078402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.078367 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 17:32:20.082401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.082377 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2b49df-66746657c5-m86lg"] Nov 03 17:32:20.118791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.118748 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 17:32:20.125206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.125172 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2b49df-b485ddf67-59l6h"] Nov 03 17:32:20.790291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.790256 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" path="/var/lib/kubelet/pods/8e0014c4-3df4-4657-8350-9db0c4ad78c6/volumes" Nov 03 17:32:20.790792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.790775 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" path="/var/lib/kubelet/pods/a6bdf97b-8183-4878-ac24-9c88fd170cdb/volumes" Nov 03 17:32:20.791199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:20.791186 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a86608-4201-4815-b4c8-dff750c2e0d1" path="/var/lib/kubelet/pods/c2a86608-4201-4815-b4c8-dff750c2e0d1/volumes" Nov 03 17:32:21.307892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.307859 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt"] Nov 03 17:32:21.308618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308597 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:21.308618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308615 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308628 6001 cpu_manager.go:414] "containerMap: removing container" podUID="068415a3-cd0f-4b74-9c9c-6a1684da815d" containerName="revoke-keys" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308633 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068415a3-cd0f-4b74-9c9c-6a1684da815d" containerName="revoke-keys" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308646 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308651 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308691 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308697 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308795 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6bdf97b-8183-4878-ac24-9c88fd170cdb" containerName="payload-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308811 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308817 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0014c4-3df4-4657-8350-9db0c4ad78c6" containerName="payload-pre-processing" Nov 03 17:32:21.308839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.308831 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="068415a3-cd0f-4b74-9c9c-6a1684da815d" containerName="revoke-keys" Nov 03 17:32:21.314058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.314035 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.330323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.330297 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:32:21.342401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.342372 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:32:21.342580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.342470 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:32:21.342580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.342506 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:32:21.347076 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.347048 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt"] Nov 03 17:32:21.379724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.379686 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.379981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.379961 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtr4h\" (UniqueName: \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.380060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.380025 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.481077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.481042 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtr4h\" (UniqueName: \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.481259 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.481093 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.481259 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.481139 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.481809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.481785 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.483584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.483564 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.516561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.516529 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtr4h\" (UniqueName: \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\") pod \"maas-api-revoke-sub-803a5dd4-48vgt\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.630404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.630316 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:21.939795 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:32:21.939756 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70e316f_065c_457d_8623_9e4e892f7eab.slice/crio-d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a WatchSource:0}: Error finding container d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a: Status 404 returned error can't find the container with id d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a Nov 03 17:32:21.941509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.941365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt"] Nov 03 17:32:21.952307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.952281 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 17:32:21.959099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.959070 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" containerID="cri-o://504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01" gracePeriod=45 Nov 03 17:32:21.959233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.959208 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" containerID="cri-o://7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509" gracePeriod=45 Nov 03 17:32:21.969863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.969824 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" containerID="cri-o://515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d" gracePeriod=45 Nov 03 17:32:21.987860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:21.987828 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 17:32:22.050174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.050127 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused" Nov 03 17:32:22.167194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.167153 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.88:8000/ready\": dial tcp 10.134.0.88:8000: connect: connection refused" Nov 03 17:32:22.284010 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:22.283971 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:32:22.403070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.403039 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:32:22.420774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.417711 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:32:22.423774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.423747 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:32:22.511839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.511752 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.511839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.511828 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.511882 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\" (UniqueName: \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.512095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.511939 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.511973 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.512095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512009 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.512095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512071 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\" (UniqueName: \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512105 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\" (UniqueName: \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512163 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512227 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512298 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512305 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.512380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512347 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512393 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\") pod \"1ffda108-c908-449e-a944-ff58dfc95125\" (UID: \"1ffda108-c908-449e-a944-ff58dfc95125\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512547 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\" (UniqueName: \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512631 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\") pod \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\" (UID: \"9afaa6bd-8c35-426f-9ba9-227a57feaf3f\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512663 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512705 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\") pod \"d24aba52-e346-44d7-a9b7-1d523e330b71\" (UID: \"d24aba52-e346-44d7-a9b7-1d523e330b71\") " Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512744 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.512821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512752 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.513288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.512939 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.513288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513165 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.513288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513229 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513360 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513559 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513580 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513593 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513606 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513618 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513630 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.513643 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.514606 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.514825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.514704 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.518978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.518943 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:22.525833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.525795 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.525833 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.525806 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:22.526020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.525852 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "kube-api-access-b4v2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:22.526020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.525922 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:22.526020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.525951 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.526192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.526027 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm" pod "1ffda108-c908-449e-a944-ff58dfc95125" (UID: "1ffda108-c908-449e-a944-ff58dfc95125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.526192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.526095 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6" pod "9afaa6bd-8c35-426f-9ba9-227a57feaf3f" (UID: "9afaa6bd-8c35-426f-9ba9-227a57feaf3f"). InnerVolumeSpecName "kube-api-access-vwxm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:22.527390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.527368 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7" pod "d24aba52-e346-44d7-a9b7-1d523e330b71" (UID: "d24aba52-e346-44d7-a9b7-1d523e330b71"). InnerVolumeSpecName "kube-api-access-jdsq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:22.615158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615119 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24aba52-e346-44d7-a9b7-1d523e330b71-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615150 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615160 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615168 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615179 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxm6\" (UniqueName: \"kubernetes.io/projected/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-kube-api-access-vwxm6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615188 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9afaa6bd-8c35-426f-9ba9-227a57feaf3f-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615197 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4v2p\" (UniqueName: \"kubernetes.io/projected/1ffda108-c908-449e-a944-ff58dfc95125-kube-api-access-b4v2p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615206 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdsq7\" (UniqueName: \"kubernetes.io/projected/d24aba52-e346-44d7-a9b7-1d523e330b71-kube-api-access-jdsq7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615224 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24aba52-e346-44d7-a9b7-1d523e330b71-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615233 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ffda108-c908-449e-a944-ff58dfc95125-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.615592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.615241 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ffda108-c908-449e-a944-ff58dfc95125-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.760431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.760392 6001 generic.go:356] "Generic (PLEG): container finished" podID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerID="504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01" exitCode=0 Nov 03 17:32:22.760431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.760482 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" Nov 03 17:32:22.760431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.760490 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" event={"ID":"9afaa6bd-8c35-426f-9ba9-227a57feaf3f","Type":"ContainerDied","Data":"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01"} Nov 03 17:32:22.760822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.760552 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm" event={"ID":"9afaa6bd-8c35-426f-9ba9-227a57feaf3f","Type":"ContainerDied","Data":"4b59ff35ec8f18ce1f7ea83cb3c0e444be83889b5d3ccf0bfe94f78e7430e5a5"} Nov 03 17:32:22.760822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.760578 6001 scope.go:122] "RemoveContainer" containerID="504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01" Nov 03 17:32:22.763030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.763005 6001 generic.go:356] "Generic (PLEG): container finished" podID="1ffda108-c908-449e-a944-ff58dfc95125" containerID="515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d" exitCode=0 Nov 03 17:32:22.763171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.763046 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" Nov 03 17:32:22.763171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.763088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" event={"ID":"1ffda108-c908-449e-a944-ff58dfc95125","Type":"ContainerDied","Data":"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d"} Nov 03 17:32:22.763171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.763142 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2" event={"ID":"1ffda108-c908-449e-a944-ff58dfc95125","Type":"ContainerDied","Data":"3fd19143a11f35cdcbffb9523fa0d72c3f351d4d6fd702903fc1c82b7b885a03"} Nov 03 17:32:22.765541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.765514 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" event={"ID":"c70e316f-065c-457d-8623-9e4e892f7eab","Type":"ContainerStarted","Data":"70f1e16e57ec2d1434b933304fc65c7a8253ad6f885c5931c969631f1f330a5e"} Nov 03 17:32:22.765658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.765567 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" event={"ID":"c70e316f-065c-457d-8623-9e4e892f7eab","Type":"ContainerStarted","Data":"d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a"} Nov 03 17:32:22.768094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.768068 6001 generic.go:356] "Generic (PLEG): container finished" podID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerID="7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509" exitCode=0 Nov 03 17:32:22.768222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.768136 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" event={"ID":"d24aba52-e346-44d7-a9b7-1d523e330b71","Type":"ContainerDied","Data":"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509"} Nov 03 17:32:22.768222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.768163 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" event={"ID":"d24aba52-e346-44d7-a9b7-1d523e330b71","Type":"ContainerDied","Data":"4303d9156a864f6265df679fa4b4628ea6ceeb888e685f4ea708cc2e99ca298c"} Nov 03 17:32:22.768222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.768172 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q" Nov 03 17:32:22.774348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.774323 6001 scope.go:122] "RemoveContainer" containerID="504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01" Nov 03 17:32:22.775166 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:22.775141 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01\": container with ID starting with 504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01 not found: ID does not exist" containerID="504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01" Nov 03 17:32:22.775268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.775206 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01"} err="failed to get container status \"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01\": rpc error: code = NotFound desc = could not find container \"504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01\": container with ID starting with 504cd638004fe976774508753cabe76e107c2f2c26bc0a37b5d20a98c6793a01 not found: ID does not exist" Nov 03 17:32:22.775268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.775230 6001 scope.go:122] "RemoveContainer" containerID="515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d" Nov 03 17:32:22.788567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.788544 6001 scope.go:122] "RemoveContainer" containerID="515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d" Nov 03 17:32:22.789182 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:22.789158 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d\": container with ID starting with 515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d not found: ID does not exist" containerID="515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d" Nov 03 17:32:22.789280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.789196 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d"} err="failed to get container status \"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d\": rpc error: code = NotFound desc = could not find container \"515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d\": container with ID starting with 515c3c6f8ae1131face8755c61d6d28eef60b344614cac34f8d282cc109aa31d not found: ID does not exist" Nov 03 17:32:22.789280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.789224 6001 scope.go:122] "RemoveContainer" containerID="7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509" Nov 03 17:32:22.805668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.805636 6001 scope.go:122] "RemoveContainer" containerID="7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509" Nov 03 17:32:22.806023 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:22.805995 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509\": container with ID starting with 7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509 not found: ID does not exist" containerID="7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509" Nov 03 17:32:22.806134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.806030 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509"} err="failed to get container status \"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509\": rpc error: code = NotFound desc = could not find container \"7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509\": container with ID starting with 7e9096fd7c7460cb7d356da874120a595a59a8d3689bcd5e7bc7f56f76b72509 not found: ID does not exist" Nov 03 17:32:22.835278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.835244 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 17:32:22.838181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.838158 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/simulator-w3-3a2fa2-kserve-755db874bc-p5zl2"] Nov 03 17:32:22.879160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.879104 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" podStartSLOduration=1.879089714 podStartE2EDuration="1.879089714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:22.87726121 +0000 UTC m=+4638.210474780" watchObservedRunningTime="2026-11-03 17:32:22.879089714 +0000 UTC m=+4638.212303282" Nov 03 17:32:22.915984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.915948 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 17:32:22.920316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.920289 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-2-w3-3a2fa2-kserve-cc5468bcd-4cpmm"] Nov 03 17:32:22.956859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.956828 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 17:32:22.967247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:22.967218 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/distinct-w3-3a2fa2-kserve-6f7d95b565-rlg6q"] Nov 03 17:32:23.780056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:23.780021 6001 generic.go:356] "Generic (PLEG): container finished" podID="c70e316f-065c-457d-8623-9e4e892f7eab" containerID="70f1e16e57ec2d1434b933304fc65c7a8253ad6f885c5931c969631f1f330a5e" exitCode=0 Nov 03 17:32:23.780275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:23.780107 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" event={"ID":"c70e316f-065c-457d-8623-9e4e892f7eab","Type":"ContainerDied","Data":"70f1e16e57ec2d1434b933304fc65c7a8253ad6f885c5931c969631f1f330a5e"} Nov 03 17:32:24.791829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:24.791797 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffda108-c908-449e-a944-ff58dfc95125" path="/var/lib/kubelet/pods/1ffda108-c908-449e-a944-ff58dfc95125/volumes" Nov 03 17:32:24.792358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:24.792343 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" path="/var/lib/kubelet/pods/9afaa6bd-8c35-426f-9ba9-227a57feaf3f/volumes" Nov 03 17:32:24.792844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:24.792829 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" path="/var/lib/kubelet/pods/d24aba52-e346-44d7-a9b7-1d523e330b71/volumes" Nov 03 17:32:24.960052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:24.960018 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:25.054560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.054430 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\") pod \"c70e316f-065c-457d-8623-9e4e892f7eab\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " Nov 03 17:32:25.054726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.054640 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\" (UniqueName: \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\") pod \"c70e316f-065c-457d-8623-9e4e892f7eab\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " Nov 03 17:32:25.054790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.054726 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\") pod \"c70e316f-065c-457d-8623-9e4e892f7eab\" (UID: \"c70e316f-065c-457d-8623-9e4e892f7eab\") " Nov 03 17:32:25.055106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.055080 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca" pod "c70e316f-065c-457d-8623-9e4e892f7eab" (UID: "c70e316f-065c-457d-8623-9e4e892f7eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:25.055236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.055216 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:25.056711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.056677 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca" pod "c70e316f-065c-457d-8623-9e4e892f7eab" (UID: "c70e316f-065c-457d-8623-9e4e892f7eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:25.056830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.056739 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h" pod "c70e316f-065c-457d-8623-9e4e892f7eab" (UID: "c70e316f-065c-457d-8623-9e4e892f7eab"). InnerVolumeSpecName "kube-api-access-mtr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:25.156335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.156297 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70e316f-065c-457d-8623-9e4e892f7eab-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:25.156335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.156330 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtr4h\" (UniqueName: \"kubernetes.io/projected/c70e316f-065c-457d-8623-9e4e892f7eab-kube-api-access-mtr4h\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:25.798153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.798119 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" Nov 03 17:32:25.798153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.798148 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt" event={"ID":"c70e316f-065c-457d-8623-9e4e892f7eab","Type":"ContainerDied","Data":"d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a"} Nov 03 17:32:25.798624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:25.798202 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3a20c218bbeef66184fca783df76158cbb29f7cb18e76b4d5405dc492b9510a" Nov 03 17:32:26.952859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:26.952785 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" containerID="cri-o://4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd" gracePeriod=25 Nov 03 17:32:26.988471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:26.988420 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" containerID="cri-o://915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58" gracePeriod=25 Nov 03 17:32:34.582703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:34.582663 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:34.583270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:34.582663 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:41.036510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:41.036470 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g"] Nov 03 17:32:41.048920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:41.048883 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8abfe23-hd24g"] Nov 03 17:32:42.791019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:42.790980 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcfc392-e362-4d8a-8f26-f63dbecb07b5" path="/var/lib/kubelet/pods/6dcfc392-e362-4d8a-8f26-f63dbecb07b5/volumes" Nov 03 17:32:44.582764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:44.582723 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:44.583201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:44.582728 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:32:45.083262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:45.083225 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 17:32:45.083572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:45.083544 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" containerID="cri-o://891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571" gracePeriod=30 Nov 03 17:32:47.008245 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:47.008195 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d0c403_83f2_49b3_94b3_23f05c1ef11c.slice/crio-conmon-4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:32:47.328878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.328848 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 17:32:47.332799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.332774 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 17:32:47.426880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.426847 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\" (UniqueName: \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\") pod \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " Nov 03 17:32:47.427106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.426962 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\" (UniqueName: \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\") pod \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " Nov 03 17:32:47.427193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.427175 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\") pod \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\" (UID: \"f3d0c403-83f2-49b3-94b3-23f05c1ef11c\") " Nov 03 17:32:47.427272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.427215 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\") pod \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\" (UID: \"ca2feca8-6a16-4bc0-bfa6-03800266de1e\") " Nov 03 17:32:47.427702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.427675 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume" pod "f3d0c403-83f2-49b3-94b3-23f05c1ef11c" (UID: "f3d0c403-83f2-49b3-94b3-23f05c1ef11c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:47.427852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.427785 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume" pod "ca2feca8-6a16-4bc0-bfa6-03800266de1e" (UID: "ca2feca8-6a16-4bc0-bfa6-03800266de1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:47.427911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.427858 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:47.429268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.429244 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst" pod "f3d0c403-83f2-49b3-94b3-23f05c1ef11c" (UID: "f3d0c403-83f2-49b3-94b3-23f05c1ef11c"). InnerVolumeSpecName "kube-api-access-rhbst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:47.429723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.429704 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h" pod "ca2feca8-6a16-4bc0-bfa6-03800266de1e" (UID: "ca2feca8-6a16-4bc0-bfa6-03800266de1e"). InnerVolumeSpecName "kube-api-access-cfw4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:47.529124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.529019 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2feca8-6a16-4bc0-bfa6-03800266de1e-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:47.529124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.529062 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfw4h\" (UniqueName: \"kubernetes.io/projected/ca2feca8-6a16-4bc0-bfa6-03800266de1e-kube-api-access-cfw4h\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:47.529124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.529076 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhbst\" (UniqueName: \"kubernetes.io/projected/f3d0c403-83f2-49b3-94b3-23f05c1ef11c-kube-api-access-rhbst\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:47.995890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.995854 6001 generic.go:356] "Generic (PLEG): container finished" podID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerID="915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58" exitCode=0 Nov 03 17:32:47.996102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.995912 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" Nov 03 17:32:47.996102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.996034 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" event={"ID":"ca2feca8-6a16-4bc0-bfa6-03800266de1e","Type":"ContainerDied","Data":"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58"} Nov 03 17:32:47.996102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.996088 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr" event={"ID":"ca2feca8-6a16-4bc0-bfa6-03800266de1e","Type":"ContainerDied","Data":"14231c59a3e11aebd45db079d3dad3990b018192e4d88956ab6859d532e976cc"} Nov 03 17:32:47.996297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.996133 6001 scope.go:122] "RemoveContainer" containerID="915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58" Nov 03 17:32:47.997788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.997759 6001 generic.go:356] "Generic (PLEG): container finished" podID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerID="4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd" exitCode=0 Nov 03 17:32:47.997952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.997809 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" Nov 03 17:32:47.997952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.997840 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" event={"ID":"f3d0c403-83f2-49b3-94b3-23f05c1ef11c","Type":"ContainerDied","Data":"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd"} Nov 03 17:32:47.997952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:47.997912 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6" event={"ID":"f3d0c403-83f2-49b3-94b3-23f05c1ef11c","Type":"ContainerDied","Data":"bb2e7eeed8a3a09f63da5601d21dd2b750f64490f302fb69a50e254222a60522"} Nov 03 17:32:48.024681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.023895 6001 scope.go:122] "RemoveContainer" containerID="915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58" Nov 03 17:32:48.025635 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:48.025608 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58\": container with ID starting with 915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58 not found: ID does not exist" containerID="915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58" Nov 03 17:32:48.025777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.025641 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58"} err="failed to get container status \"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58\": rpc error: code = NotFound desc = could not find container \"915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58\": container with ID starting with 915e8270d2fd6fd4397f9cfa430e3ebe0f2d8e07e5c70c55af8bb2391b1b2d58 not found: ID does not exist" Nov 03 17:32:48.025777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.025660 6001 scope.go:122] "RemoveContainer" containerID="4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd" Nov 03 17:32:48.037356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.037329 6001 scope.go:122] "RemoveContainer" containerID="4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd" Nov 03 17:32:48.037712 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:48.037686 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd\": container with ID starting with 4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd not found: ID does not exist" containerID="4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd" Nov 03 17:32:48.037815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.037720 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd"} err="failed to get container status \"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd\": rpc error: code = NotFound desc = could not find container \"4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd\": container with ID starting with 4fb78de5ed5620968dede9a4980cb2eb1c41e59750c7a896ebb596c3e8db41cd not found: ID does not exist" Nov 03 17:32:48.067203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.067169 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 17:32:48.081423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.081390 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3a2fa2-d8bb9bf67-t7wq6"] Nov 03 17:32:48.105219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.103788 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 17:32:48.106918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.106886 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-3a2fa2-66c55dd99f-nddcr"] Nov 03 17:32:48.790733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.790699 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" path="/var/lib/kubelet/pods/ca2feca8-6a16-4bc0-bfa6-03800266de1e/volumes" Nov 03 17:32:48.791207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:48.791191 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" path="/var/lib/kubelet/pods/f3d0c403-83f2-49b3-94b3-23f05c1ef11c/volumes" Nov 03 17:32:49.888428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.888397 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm"] Nov 03 17:32:49.889245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889222 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889245 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889267 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889274 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889304 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c70e316f-065c-457d-8623-9e4e892f7eab" containerName="revoke-keys" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889313 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70e316f-065c-457d-8623-9e4e892f7eab" containerName="revoke-keys" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889338 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889346 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889360 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" Nov 03 17:32:49.889369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889367 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889409 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889417 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889436 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889461 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889633 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d0c403-83f2-49b3-94b3-23f05c1ef11c" containerName="payload-processing" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889655 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70e316f-065c-457d-8623-9e4e892f7eab" containerName="revoke-keys" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889666 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afaa6bd-8c35-426f-9ba9-227a57feaf3f" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889676 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24aba52-e346-44d7-a9b7-1d523e330b71" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889691 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5abaaf-cd21-4be2-9435-53e2180a8229" containerName="istio-proxy" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889703 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffda108-c908-449e-a944-ff58dfc95125" containerName="main" Nov 03 17:32:49.889887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.889714 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2feca8-6a16-4bc0-bfa6-03800266de1e" containerName="payload-pre-processing" Nov 03 17:32:49.895000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.894977 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:49.914876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.914849 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm"] Nov 03 17:32:49.923038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.923010 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:32:49.923204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.923065 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:32:49.939292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.939266 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:32:49.939292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.939265 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:32:49.988372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.988342 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpz7\" (UniqueName: \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:49.988566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.988489 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:49.988566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:49.988529 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.100468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.096309 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpz7\" (UniqueName: \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.100468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.096493 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.100468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.096534 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.100468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.097562 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.105626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.105590 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.123032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.123000 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpz7\" (UniqueName: \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\") pod \"maas-api-revoke-sub-9d191c67-pgpcm\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.210373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.210333 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:50.353322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.353295 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 17:32:50.501712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501680 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.501893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501744 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.501893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501795 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.501893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501885 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501917 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501939 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.501958 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502026 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:50.502077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502034 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502180 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\") pod \"771146da-420a-4d6a-8261-0245e0ee27ad\" (UID: \"771146da-420a-4d6a-8261-0245e0ee27ad\") " Nov 03 17:32:50.502359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502178 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:50.502359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502296 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:50.502359 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502193 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:50.502555 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502519 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:50.502820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502796 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.502820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502817 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/771146da-420a-4d6a-8261-0245e0ee27ad-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.503046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502827 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.503046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502841 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.503046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.502854 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.504068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.504049 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:50.504257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.504237 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:50.504652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.504635 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:32:50.504722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.504669 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl" pod "771146da-420a-4d6a-8261-0245e0ee27ad" (UID: "771146da-420a-4d6a-8261-0245e0ee27ad"). InnerVolumeSpecName "kube-api-access-kjqzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:50.521104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.521082 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm"] Nov 03 17:32:50.522271 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:32:50.522239 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5d4136_a787_464e_8621_b9aebc35e22c.slice/crio-29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a WatchSource:0}: Error finding container 29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a: Status 404 returned error can't find the container with id 29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a Nov 03 17:32:50.604625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.604601 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/771146da-420a-4d6a-8261-0245e0ee27ad-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.604625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.604626 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/771146da-420a-4d6a-8261-0245e0ee27ad-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.604775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.604636 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:50.604775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:50.604645 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqzl\" (UniqueName: \"kubernetes.io/projected/771146da-420a-4d6a-8261-0245e0ee27ad-kube-api-access-kjqzl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:51.025982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.025876 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" event={"ID":"ef5d4136-a787-464e-8621-b9aebc35e22c","Type":"ContainerStarted","Data":"929a4095194709a848d3b278a801a76b5371f714832af92b481f55b9f06f78c3"} Nov 03 17:32:51.025982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.025946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" event={"ID":"ef5d4136-a787-464e-8621-b9aebc35e22c","Type":"ContainerStarted","Data":"29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a"} Nov 03 17:32:51.027357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.027329 6001 generic.go:356] "Generic (PLEG): container finished" podID="771146da-420a-4d6a-8261-0245e0ee27ad" containerID="891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571" exitCode=0 Nov 03 17:32:51.027497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.027360 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" Nov 03 17:32:51.027497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.027413 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" event={"ID":"771146da-420a-4d6a-8261-0245e0ee27ad","Type":"ContainerDied","Data":"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571"} Nov 03 17:32:51.027497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.027475 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q" event={"ID":"771146da-420a-4d6a-8261-0245e0ee27ad","Type":"ContainerDied","Data":"81bf4187a0dd10566dddf98894d53d8da8426389c7397a729838d06bab5e505e"} Nov 03 17:32:51.027685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.027503 6001 scope.go:122] "RemoveContainer" containerID="891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571" Nov 03 17:32:51.038357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.038341 6001 scope.go:122] "RemoveContainer" containerID="891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571" Nov 03 17:32:51.038644 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:51.038618 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571\": container with ID starting with 891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571 not found: ID does not exist" containerID="891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571" Nov 03 17:32:51.038718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.038654 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571"} err="failed to get container status \"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571\": rpc error: code = NotFound desc = could not find container \"891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571\": container with ID starting with 891283260fc7a24ba8028b75c07c6f66dc8020f8c19fb5e8f86081c058a74571 not found: ID does not exist" Nov 03 17:32:51.069044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.068986 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" podStartSLOduration=2.068971595 podStartE2EDuration="2.068971595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:32:51.068697142 +0000 UTC m=+4666.401910710" watchObservedRunningTime="2026-11-03 17:32:51.068971595 +0000 UTC m=+4666.402185165" Nov 03 17:32:51.110424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.110387 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 17:32:51.122543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:51.122511 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-5bc6q"] Nov 03 17:32:52.038207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:52.038177 6001 generic.go:356] "Generic (PLEG): container finished" podID="ef5d4136-a787-464e-8621-b9aebc35e22c" containerID="929a4095194709a848d3b278a801a76b5371f714832af92b481f55b9f06f78c3" exitCode=0 Nov 03 17:32:52.038625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:52.038217 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" event={"ID":"ef5d4136-a787-464e-8621-b9aebc35e22c","Type":"ContainerDied","Data":"929a4095194709a848d3b278a801a76b5371f714832af92b481f55b9f06f78c3"} Nov 03 17:32:52.251474 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:32:52.251412 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:32:52.789607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:52.789574 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" path="/var/lib/kubelet/pods/771146da-420a-4d6a-8261-0245e0ee27ad/volumes" Nov 03 17:32:53.211531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.211508 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:53.238890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.238857 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\") pod \"ef5d4136-a787-464e-8621-b9aebc35e22c\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " Nov 03 17:32:53.239039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.239019 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\" (UniqueName: \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\") pod \"ef5d4136-a787-464e-8621-b9aebc35e22c\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " Nov 03 17:32:53.239126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.239109 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\") pod \"ef5d4136-a787-464e-8621-b9aebc35e22c\" (UID: \"ef5d4136-a787-464e-8621-b9aebc35e22c\") " Nov 03 17:32:53.239266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.239244 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca" pod "ef5d4136-a787-464e-8621-b9aebc35e22c" (UID: "ef5d4136-a787-464e-8621-b9aebc35e22c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:53.239626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.239605 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:53.241085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.241053 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca" pod "ef5d4136-a787-464e-8621-b9aebc35e22c" (UID: "ef5d4136-a787-464e-8621-b9aebc35e22c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:53.241175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.241130 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7" pod "ef5d4136-a787-464e-8621-b9aebc35e22c" (UID: "ef5d4136-a787-464e-8621-b9aebc35e22c"). InnerVolumeSpecName "kube-api-access-xkpz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:53.341082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.340988 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkpz7\" (UniqueName: \"kubernetes.io/projected/ef5d4136-a787-464e-8621-b9aebc35e22c-kube-api-access-xkpz7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:53.341082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:53.341026 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef5d4136-a787-464e-8621-b9aebc35e22c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:32:54.052719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:54.052684 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" event={"ID":"ef5d4136-a787-464e-8621-b9aebc35e22c","Type":"ContainerDied","Data":"29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a"} Nov 03 17:32:54.052719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:54.052705 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm" Nov 03 17:32:54.052966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:32:54.052743 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29cab7b0a7254891be33be5eaf4c47bae05cffbef023e348ec9e392c8fb8c32a" Nov 03 17:33:00.231114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231076 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z"] Nov 03 17:33:00.231733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231718 6001 cpu_manager.go:414] "containerMap: removing container" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" Nov 03 17:33:00.231791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231734 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" Nov 03 17:33:00.231791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231755 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ef5d4136-a787-464e-8621-b9aebc35e22c" containerName="revoke-keys" Nov 03 17:33:00.231791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231761 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5d4136-a787-464e-8621-b9aebc35e22c" containerName="revoke-keys" Nov 03 17:33:00.231898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231882 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="771146da-420a-4d6a-8261-0245e0ee27ad" containerName="istio-proxy" Nov 03 17:33:00.231898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.231893 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5d4136-a787-464e-8621-b9aebc35e22c" containerName="revoke-keys" Nov 03 17:33:00.235330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.235307 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.255982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.255956 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z"] Nov 03 17:33:00.264876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.264847 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:33:00.265053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.264901 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:33:00.265053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.264944 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.265201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.264984 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.353860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.353826 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.354053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.353955 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn8j\" (UniqueName: \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.354126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.354053 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.455196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.455156 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.455488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.455277 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn8j\" (UniqueName: \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.455488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.455417 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.456119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.456094 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.457681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.457655 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.481553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.481468 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn8j\" (UniqueName: \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\") pod \"maas-api-revoke-sub-3024a083-9fd5z\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.549792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.549751 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:00.866641 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:00.866613 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z"] Nov 03 17:33:00.867352 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:33:00.867328 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7891b4dd_bf08_48f8_ab54_45b3ebf7d6a0.slice/crio-4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5 WatchSource:0}: Error finding container 4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5: Status 404 returned error can't find the container with id 4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5 Nov 03 17:33:01.106470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:01.106405 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" event={"ID":"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0","Type":"ContainerStarted","Data":"4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5"} Nov 03 17:33:02.116021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:02.115989 6001 generic.go:356] "Generic (PLEG): container finished" podID="7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" containerID="111dcc2004f25e3aa56d43f4602e94900c652634eaa2dfdb2cc69adb82014096" exitCode=0 Nov 03 17:33:02.116416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:02.116076 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" event={"ID":"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0","Type":"ContainerDied","Data":"111dcc2004f25e3aa56d43f4602e94900c652634eaa2dfdb2cc69adb82014096"} Nov 03 17:33:03.287239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.287212 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:03.398965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.398876 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\") pod \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " Nov 03 17:33:03.398965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.398938 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\") pod \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " Nov 03 17:33:03.399201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.398988 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\" (UniqueName: \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\") pod \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\" (UID: \"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0\") " Nov 03 17:33:03.399325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.399288 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca" pod "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" (UID: "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:33:03.399526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.399504 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:03.401252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.401223 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j" pod "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" (UID: "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0"). InnerVolumeSpecName "kube-api-access-jnn8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:03.401352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.401272 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca" pod "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" (UID: "7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:33:03.501335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.501294 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnn8j\" (UniqueName: \"kubernetes.io/projected/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-kube-api-access-jnn8j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:03.501335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:03.501332 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:04.131561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:04.131529 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" Nov 03 17:33:04.131561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:04.131549 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z" event={"ID":"7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0","Type":"ContainerDied","Data":"4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5"} Nov 03 17:33:04.131796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:04.131599 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4224e9c9df1c94d97c3ff32402ac4f0bb205d130326916d1e1ee6c666cdc18a5" Nov 03 17:33:11.056963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.056911 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf"] Nov 03 17:33:11.057657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.057641 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" containerName="revoke-keys" Nov 03 17:33:11.057709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.057658 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" containerName="revoke-keys" Nov 03 17:33:11.057779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.057769 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" containerName="revoke-keys" Nov 03 17:33:11.061648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.061627 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.078097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.078071 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:33:11.090825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.090791 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:33:11.097910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.096963 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf"] Nov 03 17:33:11.098118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.097046 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:33:11.098118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.097824 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:33:11.112974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.112937 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.112974 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.112975 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjj4t\" (UniqueName: \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.113216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.113001 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.214293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.214248 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.214590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.214571 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.214699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.214605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjj4t\" (UniqueName: \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.215229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.215207 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.216928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.216909 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.239277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.239238 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjj4t\" (UniqueName: \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\") pod \"maas-api-revoke-sub-fb1c0412-pgnkf\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.380668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.380576 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:11.789468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:11.789172 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf"] Nov 03 17:33:11.799910 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:33:11.796085 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a97d99_587b_492a_a958_384284488439.slice/crio-2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1 WatchSource:0}: Error finding container 2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1: Status 404 returned error can't find the container with id 2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1 Nov 03 17:33:12.198034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:12.197991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" event={"ID":"b6a97d99-587b-492a-a958-384284488439","Type":"ContainerStarted","Data":"7ff539b16bd419b5544354a203d92d88af9bdd83d0834aeda959f9a8f9df85e3"} Nov 03 17:33:12.198576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:12.198194 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" event={"ID":"b6a97d99-587b-492a-a958-384284488439","Type":"ContainerStarted","Data":"2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1"} Nov 03 17:33:12.253367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:12.253316 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" podStartSLOduration=1.253302281 podStartE2EDuration="1.253302281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:12.251272744 +0000 UTC m=+4687.584486312" watchObservedRunningTime="2026-11-03 17:33:12.253302281 +0000 UTC m=+4687.586515871" Nov 03 17:33:13.206476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:13.206346 6001 generic.go:356] "Generic (PLEG): container finished" podID="b6a97d99-587b-492a-a958-384284488439" containerID="7ff539b16bd419b5544354a203d92d88af9bdd83d0834aeda959f9a8f9df85e3" exitCode=0 Nov 03 17:33:13.206476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:13.206438 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" event={"ID":"b6a97d99-587b-492a-a958-384284488439","Type":"ContainerDied","Data":"7ff539b16bd419b5544354a203d92d88af9bdd83d0834aeda959f9a8f9df85e3"} Nov 03 17:33:13.679970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:13.679935 6001 scope.go:122] "RemoveContainer" containerID="ee60c851dfc48497eff2a5fb4b7dd2b5fae1975cf98576a0642ef3200b4d7d64" Nov 03 17:33:14.366404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.366371 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:14.459982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.459942 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\") pod \"b6a97d99-587b-492a-a958-384284488439\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " Nov 03 17:33:14.460164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.460044 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\") pod \"b6a97d99-587b-492a-a958-384284488439\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " Nov 03 17:33:14.460164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.460067 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\" (UniqueName: \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\") pod \"b6a97d99-587b-492a-a958-384284488439\" (UID: \"b6a97d99-587b-492a-a958-384284488439\") " Nov 03 17:33:14.460395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.460365 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca" pod "b6a97d99-587b-492a-a958-384284488439" (UID: "b6a97d99-587b-492a-a958-384284488439"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:33:14.460711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.460683 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6a97d99-587b-492a-a958-384284488439-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:14.462267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.462243 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t" pod "b6a97d99-587b-492a-a958-384284488439" (UID: "b6a97d99-587b-492a-a958-384284488439"). InnerVolumeSpecName "kube-api-access-pjj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:14.462382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.462312 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca" pod "b6a97d99-587b-492a-a958-384284488439" (UID: "b6a97d99-587b-492a-a958-384284488439"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:33:14.562090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.562039 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6a97d99-587b-492a-a958-384284488439-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:14.562090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:14.562079 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjj4t\" (UniqueName: \"kubernetes.io/projected/b6a97d99-587b-492a-a958-384284488439-kube-api-access-pjj4t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:15.221125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:15.221035 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" Nov 03 17:33:15.221125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:15.221062 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf" event={"ID":"b6a97d99-587b-492a-a958-384284488439","Type":"ContainerDied","Data":"2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1"} Nov 03 17:33:15.221125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:15.221110 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2940fb92bf0d7d6e6c8003a458574d083a2e49ac219b53df24e73a76cbdbfae1" Nov 03 17:33:21.606210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.606112 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg"] Nov 03 17:33:21.607040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.607016 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b6a97d99-587b-492a-a958-384284488439" containerName="revoke-keys" Nov 03 17:33:21.607167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.607042 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a97d99-587b-492a-a958-384284488439" containerName="revoke-keys" Nov 03 17:33:21.607244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.607234 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a97d99-587b-492a-a958-384284488439" containerName="revoke-keys" Nov 03 17:33:21.610953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.610928 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.639455 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.639401 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:33:21.639663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.639564 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:33:21.639663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.639617 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:33:21.639811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.639702 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:33:21.645016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.644982 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg"] Nov 03 17:33:21.759852 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.759807 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghm2f\" (UniqueName: \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.760066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.759873 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.760119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.760074 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.861702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.861605 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.861891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.861781 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.861965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.861909 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghm2f\" (UniqueName: \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.862364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.862343 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.864197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.864179 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.888422 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.888384 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghm2f\" (UniqueName: \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\") pod \"maas-api-revoke-sub-493b4e6d-vn5rg\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:21.925621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:21.925591 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:22.254161 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:33:22.254128 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868453dd_ed95_4ce3_b8e4_d809021c2c40.slice/crio-149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6 WatchSource:0}: Error finding container 149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6: Status 404 returned error can't find the container with id 149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6 Nov 03 17:33:22.255357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:22.255329 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg"] Nov 03 17:33:22.256590 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:33:22.256566 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:33:22.283655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:22.283622 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" event={"ID":"868453dd-ed95-4ce3-b8e4-d809021c2c40","Type":"ContainerStarted","Data":"149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6"} Nov 03 17:33:23.292931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:23.292894 6001 generic.go:356] "Generic (PLEG): container finished" podID="868453dd-ed95-4ce3-b8e4-d809021c2c40" containerID="57a6d745077403fc8f222dd60b69de64e97955d2c70f6c2be7682212244c8e20" exitCode=0 Nov 03 17:33:23.293397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:23.292964 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" event={"ID":"868453dd-ed95-4ce3-b8e4-d809021c2c40","Type":"ContainerDied","Data":"57a6d745077403fc8f222dd60b69de64e97955d2c70f6c2be7682212244c8e20"} Nov 03 17:33:24.468531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.468500 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:24.600709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.600598 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\") pod \"868453dd-ed95-4ce3-b8e4-d809021c2c40\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " Nov 03 17:33:24.600911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.600815 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\" (UniqueName: \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\") pod \"868453dd-ed95-4ce3-b8e4-d809021c2c40\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " Nov 03 17:33:24.600978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.600919 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\") pod \"868453dd-ed95-4ce3-b8e4-d809021c2c40\" (UID: \"868453dd-ed95-4ce3-b8e4-d809021c2c40\") " Nov 03 17:33:24.601031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.600988 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca" pod "868453dd-ed95-4ce3-b8e4-d809021c2c40" (UID: "868453dd-ed95-4ce3-b8e4-d809021c2c40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:33:24.601484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.601422 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:24.602976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.602951 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca" pod "868453dd-ed95-4ce3-b8e4-d809021c2c40" (UID: "868453dd-ed95-4ce3-b8e4-d809021c2c40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:33:24.603168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.603147 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f" pod "868453dd-ed95-4ce3-b8e4-d809021c2c40" (UID: "868453dd-ed95-4ce3-b8e4-d809021c2c40"). InnerVolumeSpecName "kube-api-access-ghm2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:33:24.702948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.702900 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghm2f\" (UniqueName: \"kubernetes.io/projected/868453dd-ed95-4ce3-b8e4-d809021c2c40-kube-api-access-ghm2f\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:24.702948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:24.702940 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/868453dd-ed95-4ce3-b8e4-d809021c2c40-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:33:25.309138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:25.309104 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" Nov 03 17:33:25.309138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:25.309128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg" event={"ID":"868453dd-ed95-4ce3-b8e4-d809021c2c40","Type":"ContainerDied","Data":"149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6"} Nov 03 17:33:25.309423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:33:25.309180 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="149fa38e62e17be7ce07547846d9f52e290c11a3c62a35611fee81032fab92d6" Nov 03 17:33:52.252020 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:33:52.251972 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:34:13.753624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:13.753585 6001 scope.go:122] "RemoveContainer" containerID="df66e99fad6866e20cc40cc1e6cef4eddf744d9bd95e9bbd06cb09846fec103e" Nov 03 17:34:13.787224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:13.783299 6001 scope.go:122] "RemoveContainer" containerID="54cbecd24499dc5a679951fcdbb47d626121692266cf1c23694806d97549ae51" Nov 03 17:34:19.122744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.122703 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx"] Nov 03 17:34:19.123612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.123587 6001 cpu_manager.go:414] "containerMap: removing container" podUID="868453dd-ed95-4ce3-b8e4-d809021c2c40" containerName="revoke-keys" Nov 03 17:34:19.123612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.123611 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="868453dd-ed95-4ce3-b8e4-d809021c2c40" containerName="revoke-keys" Nov 03 17:34:19.123835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.123821 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="868453dd-ed95-4ce3-b8e4-d809021c2c40" containerName="revoke-keys" Nov 03 17:34:19.128964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.128516 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.149416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.149382 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx"] Nov 03 17:34:19.156672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.156635 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:34:19.156672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.156664 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:34:19.156939 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.156732 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:34:19.158661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.158642 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:34:19.185787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.185750 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.185992 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.185898 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.186043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.185983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gv6\" (UniqueName: \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.287404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.287362 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.287649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.287483 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gv6\" (UniqueName: \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.287649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.287570 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.288160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.288121 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.290024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.289994 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.311976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.311948 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gv6\" (UniqueName: \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\") pod \"maas-api-revoke-sub-5ba88695-q6tnx\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.444601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.444501 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:19.573115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.573076 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd"] Nov 03 17:34:19.577718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.577693 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.592057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.592018 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7p85\" (UniqueName: \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.592263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.592170 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.592337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.592310 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.598749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.598718 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd"] Nov 03 17:34:19.694119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.694082 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7p85\" (UniqueName: \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.694347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.694177 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.694347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.694246 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.695014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.694953 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.696730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.696705 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.717634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.717603 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7p85\" (UniqueName: \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\") pod \"maas-api-revoke-sub-ac958a16-l45zd\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:19.754051 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:34:19.754020 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e1aec0_61f1_4c17_9ded_60cc47c994a5.slice/crio-4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0 WatchSource:0}: Error finding container 4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0: Status 404 returned error can't find the container with id 4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0 Nov 03 17:34:19.754499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.754428 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx"] Nov 03 17:34:19.894013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:19.893968 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:20.232457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.232380 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd"] Nov 03 17:34:20.232457 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:34:20.232401 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b5ef8a_8732_4972_ba05_75c931862789.slice/crio-433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489 WatchSource:0}: Error finding container 433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489: Status 404 returned error can't find the container with id 433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489 Nov 03 17:34:20.733070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.733031 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" event={"ID":"28b5ef8a-8732-4972-ba05-75c931862789","Type":"ContainerStarted","Data":"c92489003103a00bf2a2d7b4197fc8a16d67da9859f6825839b14f7d79b426e6"} Nov 03 17:34:20.733261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.733102 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" event={"ID":"28b5ef8a-8732-4972-ba05-75c931862789","Type":"ContainerStarted","Data":"433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489"} Nov 03 17:34:20.734825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.734792 6001 generic.go:356] "Generic (PLEG): container finished" podID="f8e1aec0-61f1-4c17-9ded-60cc47c994a5" containerID="6593cc0ee3921b99160b46961034c79b353edd6531841f82a28c7fcb929231de" exitCode=0 Nov 03 17:34:20.734964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.734934 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" event={"ID":"f8e1aec0-61f1-4c17-9ded-60cc47c994a5","Type":"ContainerDied","Data":"6593cc0ee3921b99160b46961034c79b353edd6531841f82a28c7fcb929231de"} Nov 03 17:34:20.735023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.734975 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" event={"ID":"f8e1aec0-61f1-4c17-9ded-60cc47c994a5","Type":"ContainerStarted","Data":"4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0"} Nov 03 17:34:20.779984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:20.779927 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" podStartSLOduration=1.779908275 podStartE2EDuration="1.779908275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:20.77656942 +0000 UTC m=+4756.109782989" watchObservedRunningTime="2026-11-03 17:34:20.779908275 +0000 UTC m=+4756.113121843" Nov 03 17:34:21.743489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:21.743425 6001 generic.go:356] "Generic (PLEG): container finished" podID="28b5ef8a-8732-4972-ba05-75c931862789" containerID="c92489003103a00bf2a2d7b4197fc8a16d67da9859f6825839b14f7d79b426e6" exitCode=0 Nov 03 17:34:21.743489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:21.743476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" event={"ID":"28b5ef8a-8732-4972-ba05-75c931862789","Type":"ContainerDied","Data":"c92489003103a00bf2a2d7b4197fc8a16d67da9859f6825839b14f7d79b426e6"} Nov 03 17:34:21.934718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:21.934687 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:22.027936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.027848 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\") pod \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " Nov 03 17:34:22.028117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.027999 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\") pod \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " Nov 03 17:34:22.028117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.028050 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\" (UniqueName: \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\") pod \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\" (UID: \"f8e1aec0-61f1-4c17-9ded-60cc47c994a5\") " Nov 03 17:34:22.028278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.028248 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca" pod "f8e1aec0-61f1-4c17-9ded-60cc47c994a5" (UID: "f8e1aec0-61f1-4c17-9ded-60cc47c994a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:22.028414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.028395 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:22.030233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.030203 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca" pod "f8e1aec0-61f1-4c17-9ded-60cc47c994a5" (UID: "f8e1aec0-61f1-4c17-9ded-60cc47c994a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:22.030367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.030319 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6" pod "f8e1aec0-61f1-4c17-9ded-60cc47c994a5" (UID: "f8e1aec0-61f1-4c17-9ded-60cc47c994a5"). InnerVolumeSpecName "kube-api-access-k7gv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:22.130004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.129960 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7gv6\" (UniqueName: \"kubernetes.io/projected/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-kube-api-access-k7gv6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:22.130004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.129997 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8e1aec0-61f1-4c17-9ded-60cc47c994a5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:22.251710 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:34:22.251671 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:34:22.756793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.756762 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" Nov 03 17:34:22.756793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.756792 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx" event={"ID":"f8e1aec0-61f1-4c17-9ded-60cc47c994a5","Type":"ContainerDied","Data":"4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0"} Nov 03 17:34:22.757381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.756852 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4962d5cce0ad4246e76f51505c5bbec07746eb4fa349ce11a8c19475d470dbd0" Nov 03 17:34:22.899894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.899869 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:22.940810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.940783 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\" (UniqueName: \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\") pod \"28b5ef8a-8732-4972-ba05-75c931862789\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " Nov 03 17:34:22.941043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.940894 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\") pod \"28b5ef8a-8732-4972-ba05-75c931862789\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " Nov 03 17:34:22.941152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.941134 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\") pod \"28b5ef8a-8732-4972-ba05-75c931862789\" (UID: \"28b5ef8a-8732-4972-ba05-75c931862789\") " Nov 03 17:34:22.941740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.941704 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca" pod "28b5ef8a-8732-4972-ba05-75c931862789" (UID: "28b5ef8a-8732-4972-ba05-75c931862789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:22.943795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.943763 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca" pod "28b5ef8a-8732-4972-ba05-75c931862789" (UID: "28b5ef8a-8732-4972-ba05-75c931862789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:22.943922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:22.943863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85" pod "28b5ef8a-8732-4972-ba05-75c931862789" (UID: "28b5ef8a-8732-4972-ba05-75c931862789"). InnerVolumeSpecName "kube-api-access-l7p85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:23.032617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.032514 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7"] Nov 03 17:34:23.035869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.035834 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9dca08c-s5qv7"] Nov 03 17:34:23.043584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.043544 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:23.043584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.043577 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7p85\" (UniqueName: \"kubernetes.io/projected/28b5ef8a-8732-4972-ba05-75c931862789-kube-api-access-l7p85\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:23.043584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.043589 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b5ef8a-8732-4972-ba05-75c931862789-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:23.766395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.766362 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" Nov 03 17:34:23.766880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.766393 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd" event={"ID":"28b5ef8a-8732-4972-ba05-75c931862789","Type":"ContainerDied","Data":"433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489"} Nov 03 17:34:23.766880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:23.766466 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433bf1b3a48f14befa33919ad89c1445d3b42e809d738991aab9f4201c5d5489" Nov 03 17:34:24.790788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:24.790743 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacd2345-cd1c-4d5c-a3f7-b4e55320efcb" path="/var/lib/kubelet/pods/dacd2345-cd1c-4d5c-a3f7-b4e55320efcb/volumes" Nov 03 17:34:29.427668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.427619 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf"] Nov 03 17:34:29.430081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.430035 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f8e1aec0-61f1-4c17-9ded-60cc47c994a5" containerName="revoke-keys" Nov 03 17:34:29.430318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.430298 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8e1aec0-61f1-4c17-9ded-60cc47c994a5" containerName="revoke-keys" Nov 03 17:34:29.430523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.430507 6001 cpu_manager.go:414] "containerMap: removing container" podUID="28b5ef8a-8732-4972-ba05-75c931862789" containerName="revoke-keys" Nov 03 17:34:29.430640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.430627 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b5ef8a-8732-4972-ba05-75c931862789" containerName="revoke-keys" Nov 03 17:34:29.430962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.430947 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b5ef8a-8732-4972-ba05-75c931862789" containerName="revoke-keys" Nov 03 17:34:29.431119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.431104 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8e1aec0-61f1-4c17-9ded-60cc47c994a5" containerName="revoke-keys" Nov 03 17:34:29.436823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.436787 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.456212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.456179 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf"] Nov 03 17:34:29.463888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.463857 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:34:29.464082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.463895 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:34:29.464333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.464309 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:34:29.464391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.464366 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:34:29.535855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.535801 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.536064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.535862 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7mr2\" (UniqueName: \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.536064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.535963 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.637945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.637905 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.637945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.637951 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7mr2\" (UniqueName: \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.638185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.638072 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.638899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.638870 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.640430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.640403 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.662770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.662735 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7mr2\" (UniqueName: \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\") pod \"maas-api-revoke-sub-e59103c8-wmfpf\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.764326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.764290 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:29.876290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.876255 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf"] Nov 03 17:34:29.881807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.881780 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:29.901921 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.901890 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf"] Nov 03 17:34:29.943500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.943421 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74kr\" (UniqueName: \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:29.943681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.943536 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:29.943681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:29.943622 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.044666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.044584 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74kr\" (UniqueName: \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.044666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.044642 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.044900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.044746 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.045397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.045370 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.046988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.046959 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.071810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.071773 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74kr\" (UniqueName: \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\") pod \"maas-api-revoke-sub-c954b259-68znf\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.074565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.074532 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf"] Nov 03 17:34:30.079700 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:34:30.079672 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea22dca3_e226_47d6_9954_2093b9ee3e80.slice/crio-1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac WatchSource:0}: Error finding container 1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac: Status 404 returned error can't find the container with id 1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac Nov 03 17:34:30.198809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.198770 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:30.509643 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:34:30.509607 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a2be10_d0b6_4f83_bbc4_fc3d84a2a43b.slice/crio-fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf WatchSource:0}: Error finding container fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf: Status 404 returned error can't find the container with id fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf Nov 03 17:34:30.509988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.509954 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf"] Nov 03 17:34:30.822135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.822098 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" event={"ID":"ea22dca3-e226-47d6-9954-2093b9ee3e80","Type":"ContainerStarted","Data":"3581f200863f94a8711afb5bd080ce454973b4ce5cb7bc43704ff760d48e8a92"} Nov 03 17:34:30.822352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.822168 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" event={"ID":"ea22dca3-e226-47d6-9954-2093b9ee3e80","Type":"ContainerStarted","Data":"1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac"} Nov 03 17:34:30.823594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.823565 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" event={"ID":"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b","Type":"ContainerStarted","Data":"fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf"} Nov 03 17:34:30.876170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:30.876113 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" podStartSLOduration=1.876097309 podStartE2EDuration="1.876097309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:30.874267445 +0000 UTC m=+4766.207481014" watchObservedRunningTime="2026-11-03 17:34:30.876097309 +0000 UTC m=+4766.209310906" Nov 03 17:34:31.836500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:31.836461 6001 generic.go:356] "Generic (PLEG): container finished" podID="25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" containerID="d11f0008a2f4308cdac83b6437de5b189544cba1a37173593ec3c009810038e7" exitCode=0 Nov 03 17:34:31.836973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:31.836543 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" event={"ID":"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b","Type":"ContainerDied","Data":"d11f0008a2f4308cdac83b6437de5b189544cba1a37173593ec3c009810038e7"} Nov 03 17:34:31.838367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:31.838342 6001 generic.go:356] "Generic (PLEG): container finished" podID="ea22dca3-e226-47d6-9954-2093b9ee3e80" containerID="3581f200863f94a8711afb5bd080ce454973b4ce5cb7bc43704ff760d48e8a92" exitCode=0 Nov 03 17:34:31.838521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:31.838432 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" event={"ID":"ea22dca3-e226-47d6-9954-2093b9ee3e80","Type":"ContainerDied","Data":"3581f200863f94a8711afb5bd080ce454973b4ce5cb7bc43704ff760d48e8a92"} Nov 03 17:34:33.028493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.027550 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:33.071945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.071861 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:33.189980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.189900 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\" (UniqueName: \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\") pod \"ea22dca3-e226-47d6-9954-2093b9ee3e80\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " Nov 03 17:34:33.190139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.189995 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\") pod \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " Nov 03 17:34:33.190139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190110 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\") pod \"ea22dca3-e226-47d6-9954-2093b9ee3e80\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " Nov 03 17:34:33.190246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190154 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\") pod \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " Nov 03 17:34:33.190246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190192 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\") pod \"ea22dca3-e226-47d6-9954-2093b9ee3e80\" (UID: \"ea22dca3-e226-47d6-9954-2093b9ee3e80\") " Nov 03 17:34:33.190356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190242 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\" (UniqueName: \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\") pod \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\" (UID: \"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b\") " Nov 03 17:34:33.190544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190513 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca" pod "ea22dca3-e226-47d6-9954-2093b9ee3e80" (UID: "ea22dca3-e226-47d6-9954-2093b9ee3e80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:33.190677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190543 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca" pod "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" (UID: "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:33.191006 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.190979 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.191146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.191010 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.192330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.192294 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca" pod "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" (UID: "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:33.192431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.192351 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca" pod "ea22dca3-e226-47d6-9954-2093b9ee3e80" (UID: "ea22dca3-e226-47d6-9954-2093b9ee3e80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:33.192830 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.192811 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr" pod "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" (UID: "25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b"). InnerVolumeSpecName "kube-api-access-f74kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:33.192884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.192837 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2" pod "ea22dca3-e226-47d6-9954-2093b9ee3e80" (UID: "ea22dca3-e226-47d6-9954-2093b9ee3e80"). InnerVolumeSpecName "kube-api-access-g7mr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:33.292203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.292166 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7mr2\" (UniqueName: \"kubernetes.io/projected/ea22dca3-e226-47d6-9954-2093b9ee3e80-kube-api-access-g7mr2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.292203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.292197 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.292203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.292207 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea22dca3-e226-47d6-9954-2093b9ee3e80-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.292469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.292216 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f74kr\" (UniqueName: \"kubernetes.io/projected/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b-kube-api-access-f74kr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:34:33.856677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.856647 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" Nov 03 17:34:33.856865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.856688 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf" event={"ID":"25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b","Type":"ContainerDied","Data":"fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf"} Nov 03 17:34:33.856865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.856734 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcffc73913e2c5868b97947b78db10a14092baaa463bee7ea2a86281122f17cf" Nov 03 17:34:33.858347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.858317 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" event={"ID":"ea22dca3-e226-47d6-9954-2093b9ee3e80","Type":"ContainerDied","Data":"1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac"} Nov 03 17:34:33.858347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.858335 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf" Nov 03 17:34:33.858593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:33.858364 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db984365c42cc6c121f0315e631a2b3ee46574e0bb5a4a09408a1a8609eefac" Nov 03 17:34:52.271101 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:34:52.271019 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:34:57.057712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:34:57.057679 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:35:02.058983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:02.058921 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" containerID="cri-o://a4352befcab86d67f6b5e0f1642177d563a9152d161bf3258ca5efb64912446a" gracePeriod=25 Nov 03 17:35:03.209612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:03.209570 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:35:03.270640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:03.270591 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:35:05.819998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:05.819970 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:35:05.932927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:05.932901 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:35:06.144734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:06.144644 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:08.210539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:08.210489 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" containerID="cri-o://270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a" gracePeriod=25 Nov 03 17:35:08.271283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:08.271224 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" containerID="cri-o://57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0" gracePeriod=25 Nov 03 17:35:12.958379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:12.958331 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:35:12.988035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:12.988001 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:35:16.144691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:16.144649 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:17.358390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.358356 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w"] Nov 03 17:35:17.359130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359111 6001 cpu_manager.go:414] "containerMap: removing container" podUID="25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" containerName="revoke-keys" Nov 03 17:35:17.359178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359133 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" containerName="revoke-keys" Nov 03 17:35:17.359178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359171 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ea22dca3-e226-47d6-9954-2093b9ee3e80" containerName="revoke-keys" Nov 03 17:35:17.359178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359177 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea22dca3-e226-47d6-9954-2093b9ee3e80" containerName="revoke-keys" Nov 03 17:35:17.359277 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359274 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" containerName="revoke-keys" Nov 03 17:35:17.359317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.359287 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea22dca3-e226-47d6-9954-2093b9ee3e80" containerName="revoke-keys" Nov 03 17:35:17.364242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.364225 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.383646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.383616 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w"] Nov 03 17:35:17.392201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.392176 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:35:17.392436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.392397 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:35:17.392550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.392286 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:35:17.392550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.392184 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:35:17.404247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.404212 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:17.404406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.404384 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:17.550105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.550066 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.550372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.550348 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.550470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.550391 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52jcv\" (UniqueName: \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.651996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.651904 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.651996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.651949 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52jcv\" (UniqueName: \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.652215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.652014 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.652690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.652661 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.654345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.654316 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.677643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.677613 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52jcv\" (UniqueName: \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\") pod \"maas-api-revoke-sub-fccf1837-msm7w\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.678227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.678203 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:17.958765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.958671 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" containerID="cri-o://26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b" gracePeriod=25 Nov 03 17:35:17.984846 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:35:17.984813 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc4e99f_ab2b_4144_a4e7_d8a97537f656.slice/crio-dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7 WatchSource:0}: Error finding container dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7: Status 404 returned error can't find the container with id dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7 Nov 03 17:35:17.985131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.985107 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w"] Nov 03 17:35:17.988582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:17.988554 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" containerID="cri-o://8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571" gracePeriod=25 Nov 03 17:35:18.186197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:18.186157 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" event={"ID":"9fc4e99f-ab2b-4144-a4e7-d8a97537f656","Type":"ContainerStarted","Data":"dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7"} Nov 03 17:35:19.195116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:19.195028 6001 generic.go:356] "Generic (PLEG): container finished" podID="9fc4e99f-ab2b-4144-a4e7-d8a97537f656" containerID="c621f5557618da1cb95662436df79fe303403804c364164af2331ac1bf9efa94" exitCode=0 Nov 03 17:35:19.195520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:19.195120 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" event={"ID":"9fc4e99f-ab2b-4144-a4e7-d8a97537f656","Type":"ContainerDied","Data":"c621f5557618da1cb95662436df79fe303403804c364164af2331ac1bf9efa94"} Nov 03 17:35:20.359621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.359593 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:20.487427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.487394 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\") pod \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " Nov 03 17:35:20.487673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.487484 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\" (UniqueName: \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\") pod \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " Nov 03 17:35:20.487673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.487572 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\") pod \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\" (UID: \"9fc4e99f-ab2b-4144-a4e7-d8a97537f656\") " Nov 03 17:35:20.487817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.487771 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca" pod "9fc4e99f-ab2b-4144-a4e7-d8a97537f656" (UID: "9fc4e99f-ab2b-4144-a4e7-d8a97537f656"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:20.488184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.488162 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:20.489793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.489765 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca" pod "9fc4e99f-ab2b-4144-a4e7-d8a97537f656" (UID: "9fc4e99f-ab2b-4144-a4e7-d8a97537f656"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:20.489886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.489793 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv" pod "9fc4e99f-ab2b-4144-a4e7-d8a97537f656" (UID: "9fc4e99f-ab2b-4144-a4e7-d8a97537f656"). InnerVolumeSpecName "kube-api-access-52jcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:20.589209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.589169 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:20.589209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:20.589208 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52jcv\" (UniqueName: \"kubernetes.io/projected/9fc4e99f-ab2b-4144-a4e7-d8a97537f656-kube-api-access-52jcv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:21.217148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:21.217056 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" Nov 03 17:35:21.217148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:21.217074 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w" event={"ID":"9fc4e99f-ab2b-4144-a4e7-d8a97537f656","Type":"ContainerDied","Data":"dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7"} Nov 03 17:35:21.217377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:21.217165 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dee14f1ea8575c2a33d521118571b3b5993ae933f751443a78832e5237c26fc7" Nov 03 17:35:22.226900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.226864 6001 generic.go:356] "Generic (PLEG): container finished" podID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerID="a4352befcab86d67f6b5e0f1642177d563a9152d161bf3258ca5efb64912446a" exitCode=0 Nov 03 17:35:22.227305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.226937 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" event={"ID":"ef783c38-2ccf-4c46-80e0-8c7246752f15","Type":"ContainerDied","Data":"a4352befcab86d67f6b5e0f1642177d563a9152d161bf3258ca5efb64912446a"} Nov 03 17:35:22.261343 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:22.261305 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:35:22.338592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.338559 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:35:22.512682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.512642 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\") pod \"ef783c38-2ccf-4c46-80e0-8c7246752f15\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " Nov 03 17:35:22.512874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.512735 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\" (UniqueName: \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\") pod \"ef783c38-2ccf-4c46-80e0-8c7246752f15\" (UID: \"ef783c38-2ccf-4c46-80e0-8c7246752f15\") " Nov 03 17:35:22.513072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.513050 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume" pod "ef783c38-2ccf-4c46-80e0-8c7246752f15" (UID: "ef783c38-2ccf-4c46-80e0-8c7246752f15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:22.515068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.515045 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt" pod "ef783c38-2ccf-4c46-80e0-8c7246752f15" (UID: "ef783c38-2ccf-4c46-80e0-8c7246752f15"). InnerVolumeSpecName "kube-api-access-pb4wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:22.614475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.614354 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb4wt\" (UniqueName: \"kubernetes.io/projected/ef783c38-2ccf-4c46-80e0-8c7246752f15-kube-api-access-pb4wt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:22.614475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:22.614391 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef783c38-2ccf-4c46-80e0-8c7246752f15-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:23.235711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:23.235682 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" Nov 03 17:35:23.236236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:23.235681 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b" event={"ID":"ef783c38-2ccf-4c46-80e0-8c7246752f15","Type":"ContainerDied","Data":"140f2f0529d0458236d8820bd9ed641e21bada06718e0703cac26df2cf78672e"} Nov 03 17:35:23.236236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:23.235819 6001 scope.go:122] "RemoveContainer" containerID="a4352befcab86d67f6b5e0f1642177d563a9152d161bf3258ca5efb64912446a" Nov 03 17:35:23.290152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:23.290118 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:35:23.302338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:23.302305 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-da409b-6cdfd96bc-5972b"] Nov 03 17:35:24.789763 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:24.789732 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" path="/var/lib/kubelet/pods/ef783c38-2ccf-4c46-80e0-8c7246752f15/volumes" Nov 03 17:35:25.392149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:25.392110 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:35:25.392402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:25.392377 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" containerID="cri-o://f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68" gracePeriod=30 Nov 03 17:35:26.208092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:26.208049 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:26.208533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:26.208064 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:27.403668 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.403627 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:27.404099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.403704 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:27.767130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.767086 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6"] Nov 03 17:35:27.767827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.767805 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" Nov 03 17:35:27.767827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.767826 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" Nov 03 17:35:27.768023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.767883 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9fc4e99f-ab2b-4144-a4e7-d8a97537f656" containerName="revoke-keys" Nov 03 17:35:27.768023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.767892 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc4e99f-ab2b-4144-a4e7-d8a97537f656" containerName="revoke-keys" Nov 03 17:35:27.768141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.768044 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc4e99f-ab2b-4144-a4e7-d8a97537f656" containerName="revoke-keys" Nov 03 17:35:27.768141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.768068 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef783c38-2ccf-4c46-80e0-8c7246752f15" containerName="payload-pre-processing" Nov 03 17:35:27.772993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.772972 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.795932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.795899 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6"] Nov 03 17:35:27.804891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.804857 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:35:27.805058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.804898 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:35:27.805058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.804898 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:35:27.805058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.805019 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:35:27.879920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.879888 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrthg\" (UniqueName: \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.880104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.880054 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.880239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.880222 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.981031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.980995 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.981236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.981106 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.981236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.981222 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrthg\" (UniqueName: \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.981732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.981705 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:27.983730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:27.983703 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:28.005503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.005477 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrthg\" (UniqueName: \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\") pod \"maas-api-revoke-sub-760c3229-xbjg6\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:28.087337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.087245 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:28.416609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.416577 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6"] Nov 03 17:35:28.417709 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:35:28.417684 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319fc1ea_8e21_4421_9e32_b7e0d099b54c.slice/crio-186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1 WatchSource:0}: Error finding container 186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1: Status 404 returned error can't find the container with id 186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1 Nov 03 17:35:28.530071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.530035 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:35:28.576849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.575654 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:35:28.588806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.588780 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\") pod \"b2617b18-379e-402d-9021-42c78c68d4be\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " Nov 03 17:35:28.588989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.588927 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\") pod \"b2617b18-379e-402d-9021-42c78c68d4be\" (UID: \"b2617b18-379e-402d-9021-42c78c68d4be\") " Nov 03 17:35:28.589155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.589129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume" pod "b2617b18-379e-402d-9021-42c78c68d4be" (UID: "b2617b18-379e-402d-9021-42c78c68d4be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:28.589434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.589415 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2617b18-379e-402d-9021-42c78c68d4be-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:28.591075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.591052 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn" pod "b2617b18-379e-402d-9021-42c78c68d4be" (UID: "b2617b18-379e-402d-9021-42c78c68d4be"). InnerVolumeSpecName "kube-api-access-cwlkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:28.690388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.690362 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\" (UniqueName: \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\") pod \"d52f73af-1985-484f-9c57-a4683e472589\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " Nov 03 17:35:28.690562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.690438 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\") pod \"d52f73af-1985-484f-9c57-a4683e472589\" (UID: \"d52f73af-1985-484f-9c57-a4683e472589\") " Nov 03 17:35:28.690700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.690686 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwlkn\" (UniqueName: \"kubernetes.io/projected/b2617b18-379e-402d-9021-42c78c68d4be-kube-api-access-cwlkn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:28.690846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.690824 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume" pod "d52f73af-1985-484f-9c57-a4683e472589" (UID: "d52f73af-1985-484f-9c57-a4683e472589"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:28.692469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.692435 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg" pod "d52f73af-1985-484f-9c57-a4683e472589" (UID: "d52f73af-1985-484f-9c57-a4683e472589"). InnerVolumeSpecName "kube-api-access-65nfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:28.791837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.791807 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65nfg\" (UniqueName: \"kubernetes.io/projected/d52f73af-1985-484f-9c57-a4683e472589-kube-api-access-65nfg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:28.791837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:28.791833 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d52f73af-1985-484f-9c57-a4683e472589-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:29.287472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.287356 6001 generic.go:356] "Generic (PLEG): container finished" podID="d52f73af-1985-484f-9c57-a4683e472589" containerID="57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0" exitCode=0 Nov 03 17:35:29.287472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.287411 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" Nov 03 17:35:29.287718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.287474 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" event={"ID":"d52f73af-1985-484f-9c57-a4683e472589","Type":"ContainerDied","Data":"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0"} Nov 03 17:35:29.287718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.287538 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg" event={"ID":"d52f73af-1985-484f-9c57-a4683e472589","Type":"ContainerDied","Data":"5a8f0954f8a8be5322efea64fdf2f1d5ed2242138edadc296ebbed696df2da8e"} Nov 03 17:35:29.287718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.287570 6001 scope.go:122] "RemoveContainer" containerID="57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0" Nov 03 17:35:29.289229 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.289206 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" event={"ID":"319fc1ea-8e21-4421-9e32-b7e0d099b54c","Type":"ContainerStarted","Data":"5de2c2aab1bc652684f79af5e267051f39d244534f9ff85b6fbc3a9a65f1fd5c"} Nov 03 17:35:29.289339 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.289261 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" event={"ID":"319fc1ea-8e21-4421-9e32-b7e0d099b54c","Type":"ContainerStarted","Data":"186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1"} Nov 03 17:35:29.290799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.290775 6001 generic.go:356] "Generic (PLEG): container finished" podID="b2617b18-379e-402d-9021-42c78c68d4be" containerID="270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a" exitCode=0 Nov 03 17:35:29.290903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.290813 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" Nov 03 17:35:29.290903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.290853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" event={"ID":"b2617b18-379e-402d-9021-42c78c68d4be","Type":"ContainerDied","Data":"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a"} Nov 03 17:35:29.290903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.290892 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26" event={"ID":"b2617b18-379e-402d-9021-42c78c68d4be","Type":"ContainerDied","Data":"21c19d1853346b41a6121fd70eca224f574a8636f524c940eb30acc0a5a82feb"} Nov 03 17:35:29.299476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.299436 6001 scope.go:122] "RemoveContainer" containerID="57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0" Nov 03 17:35:29.299773 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:29.299755 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0\": container with ID starting with 57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0 not found: ID does not exist" containerID="57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0" Nov 03 17:35:29.299824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.299782 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0"} err="failed to get container status \"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0\": rpc error: code = NotFound desc = could not find container \"57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0\": container with ID starting with 57278833def27e623b00513df5434564bf6e3c4be2087e4c93920d2a367957b0 not found: ID does not exist" Nov 03 17:35:29.299824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.299801 6001 scope.go:122] "RemoveContainer" containerID="270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a" Nov 03 17:35:29.309807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.309788 6001 scope.go:122] "RemoveContainer" containerID="270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a" Nov 03 17:35:29.310067 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:29.310048 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a\": container with ID starting with 270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a not found: ID does not exist" containerID="270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a" Nov 03 17:35:29.310114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.310074 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a"} err="failed to get container status \"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a\": rpc error: code = NotFound desc = could not find container \"270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a\": container with ID starting with 270a2d8b56715e7bbd74794233bf57ac545cf84059fff795501694bc3186be1a not found: ID does not exist" Nov 03 17:35:29.345373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.345341 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:35:29.350864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.350836 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d6fcae-6f96cc485c-lnb26"] Nov 03 17:35:29.385243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.385209 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:35:29.392290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.392259 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d6fcae-6ff74cb959-7qlzg"] Nov 03 17:35:29.433844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:29.433786 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" podStartSLOduration=2.433771953 podStartE2EDuration="2.433771953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:29.431111778 +0000 UTC m=+4824.764325347" watchObservedRunningTime="2026-11-03 17:35:29.433771953 +0000 UTC m=+4824.766985521" Nov 03 17:35:30.301546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.301513 6001 generic.go:356] "Generic (PLEG): container finished" podID="319fc1ea-8e21-4421-9e32-b7e0d099b54c" containerID="5de2c2aab1bc652684f79af5e267051f39d244534f9ff85b6fbc3a9a65f1fd5c" exitCode=0 Nov 03 17:35:30.301762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.301601 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" event={"ID":"319fc1ea-8e21-4421-9e32-b7e0d099b54c","Type":"ContainerDied","Data":"5de2c2aab1bc652684f79af5e267051f39d244534f9ff85b6fbc3a9a65f1fd5c"} Nov 03 17:35:30.683857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.683824 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:35:30.719722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.719695 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.719905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.719753 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.719905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.719800 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.719905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.719839 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.719905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.719876 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.720190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720012 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.720190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720022 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:30.720190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720096 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.720190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720106 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:30.720417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720219 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.720417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720255 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\") pod \"77ed6a75-097f-471c-92f9-2356c11edb19\" (UID: \"77ed6a75-097f-471c-92f9-2356c11edb19\") " Nov 03 17:35:30.720417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720288 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:30.720631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720437 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:30.720631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720494 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:30.720989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720921 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.720989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720947 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.720989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720965 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.720989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720981 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.721193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.720996 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77ed6a75-097f-471c-92f9-2356c11edb19-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.722389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.722363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:30.722782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.722757 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:30.722888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.722842 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "kube-api-access-lp8vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:30.722937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.722911 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token" pod "77ed6a75-097f-471c-92f9-2356c11edb19" (UID: "77ed6a75-097f-471c-92f9-2356c11edb19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:30.789089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.789056 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2617b18-379e-402d-9021-42c78c68d4be" path="/var/lib/kubelet/pods/b2617b18-379e-402d-9021-42c78c68d4be/volumes" Nov 03 17:35:30.789539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.789524 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d52f73af-1985-484f-9c57-a4683e472589" path="/var/lib/kubelet/pods/d52f73af-1985-484f-9c57-a4683e472589/volumes" Nov 03 17:35:30.822056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.821982 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77ed6a75-097f-471c-92f9-2356c11edb19-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.822056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.822007 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77ed6a75-097f-471c-92f9-2356c11edb19-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.822056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.822016 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8vf\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-kube-api-access-lp8vf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:30.822056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:30.822027 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77ed6a75-097f-471c-92f9-2356c11edb19-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:31.311426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.311394 6001 generic.go:356] "Generic (PLEG): container finished" podID="77ed6a75-097f-471c-92f9-2356c11edb19" containerID="f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68" exitCode=0 Nov 03 17:35:31.311647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.311429 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerDied","Data":"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68"} Nov 03 17:35:31.311647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.311522 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" event={"ID":"77ed6a75-097f-471c-92f9-2356c11edb19","Type":"ContainerDied","Data":"a50d2f06730828c014b349f6ea8d8b8cc1ad19f631002f751939e8f737f9260d"} Nov 03 17:35:31.311647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.311545 6001 scope.go:122] "RemoveContainer" containerID="f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68" Nov 03 17:35:31.311647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.311471 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf" Nov 03 17:35:31.328468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.328417 6001 scope.go:122] "RemoveContainer" containerID="45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385" Nov 03 17:35:31.339163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.339136 6001 scope.go:122] "RemoveContainer" containerID="f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68" Nov 03 17:35:31.339535 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:31.339509 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68\": container with ID starting with f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68 not found: ID does not exist" containerID="f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68" Nov 03 17:35:31.339666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.339542 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68"} err="failed to get container status \"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68\": rpc error: code = NotFound desc = could not find container \"f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68\": container with ID starting with f48e3714690da3f3090074507de61102c4e5327cd8199e683de40666c7cc4a68 not found: ID does not exist" Nov 03 17:35:31.339666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.339561 6001 scope.go:122] "RemoveContainer" containerID="45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385" Nov 03 17:35:31.339862 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:31.339842 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385\": container with ID starting with 45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385 not found: ID does not exist" containerID="45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385" Nov 03 17:35:31.339929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.339869 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385"} err="failed to get container status \"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385\": rpc error: code = NotFound desc = could not find container \"45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385\": container with ID starting with 45bed4e08cee6f57660525545aec0d0953eb5eb852a3f788184272531bc0a385 not found: ID does not exist" Nov 03 17:35:31.370544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.370509 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:35:31.383120 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.383089 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-gvfbf"] Nov 03 17:35:31.476217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.476186 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:31.531911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.531865 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\") pod \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " Nov 03 17:35:31.532096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.531931 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\" (UniqueName: \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\") pod \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " Nov 03 17:35:31.532096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.532066 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\") pod \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\" (UID: \"319fc1ea-8e21-4421-9e32-b7e0d099b54c\") " Nov 03 17:35:31.532323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.532297 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca" pod "319fc1ea-8e21-4421-9e32-b7e0d099b54c" (UID: "319fc1ea-8e21-4421-9e32-b7e0d099b54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:31.532720 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.532693 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:31.534265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.534237 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca" pod "319fc1ea-8e21-4421-9e32-b7e0d099b54c" (UID: "319fc1ea-8e21-4421-9e32-b7e0d099b54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:31.534265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.534249 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg" pod "319fc1ea-8e21-4421-9e32-b7e0d099b54c" (UID: "319fc1ea-8e21-4421-9e32-b7e0d099b54c"). InnerVolumeSpecName "kube-api-access-hrthg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:31.634197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.634118 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/319fc1ea-8e21-4421-9e32-b7e0d099b54c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:31.634197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:31.634149 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrthg\" (UniqueName: \"kubernetes.io/projected/319fc1ea-8e21-4421-9e32-b7e0d099b54c-kube-api-access-hrthg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:32.063163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.063126 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:35:32.098631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.098573 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:35:32.357231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.357144 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" event={"ID":"319fc1ea-8e21-4421-9e32-b7e0d099b54c","Type":"ContainerDied","Data":"186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1"} Nov 03 17:35:32.357231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.357201 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186a49cde75f23b909f247ff4b30a1e77bdc2f01aba6465d540a98d9b6630fc1" Nov 03 17:35:32.357231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.357157 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6" Nov 03 17:35:32.836891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:32.836834 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" path="/var/lib/kubelet/pods/77ed6a75-097f-471c-92f9-2356c11edb19/volumes" Nov 03 17:35:34.958870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.958829 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:35:34.959378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.959102 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" containerID="cri-o://6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29" gracePeriod=30 Nov 03 17:35:34.971665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.971634 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:35:34.971891 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.971869 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" containerID="cri-o://fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd" gracePeriod=30 Nov 03 17:35:34.974993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.974967 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:35:34.975215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:34.975182 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" containerID="cri-o://90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772" gracePeriod=30 Nov 03 17:35:36.208638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:36.208592 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:36.209071 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:36.208668 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:37.063745 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:37.063694 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" containerID="cri-o://7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d" gracePeriod=25 Nov 03 17:35:37.099598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:37.099547 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" containerID="cri-o://7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e" gracePeriod=25 Nov 03 17:35:38.285326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.285300 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:35:38.403497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.399417 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:35:38.430652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.430622 6001 generic.go:356] "Generic (PLEG): container finished" podID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerID="8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571" exitCode=0 Nov 03 17:35:38.430842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.430669 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" Nov 03 17:35:38.430842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.430710 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" event={"ID":"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842","Type":"ContainerDied","Data":"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571"} Nov 03 17:35:38.430842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.430776 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5" event={"ID":"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842","Type":"ContainerDied","Data":"fc279cf35cff4cd0358c8e784c5fdb4cc6a66d552081c35d8696e7619bff17df"} Nov 03 17:35:38.430842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.430809 6001 scope.go:122] "RemoveContainer" containerID="8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571" Nov 03 17:35:38.432278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.432257 6001 generic.go:356] "Generic (PLEG): container finished" podID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerID="26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b" exitCode=0 Nov 03 17:35:38.432395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.432289 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" Nov 03 17:35:38.432395 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.432344 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" event={"ID":"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389","Type":"ContainerDied","Data":"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b"} Nov 03 17:35:38.432576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.432401 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4" event={"ID":"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389","Type":"ContainerDied","Data":"a615a5315f98a10dc59b4f1eaa43ff52c74f5252018457b075b1e6dd161b4a61"} Nov 03 17:35:38.441990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.441966 6001 scope.go:122] "RemoveContainer" containerID="8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571" Nov 03 17:35:38.442360 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:38.442338 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571\": container with ID starting with 8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571 not found: ID does not exist" containerID="8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571" Nov 03 17:35:38.442501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.442381 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571"} err="failed to get container status \"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571\": rpc error: code = NotFound desc = could not find container \"8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571\": container with ID starting with 8f63921e74b86fe0f7f715e9d6648bd9a1812b89145c016612bf0e2b7d84f571 not found: ID does not exist" Nov 03 17:35:38.442501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.442404 6001 scope.go:122] "RemoveContainer" containerID="26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b" Nov 03 17:35:38.454265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.454234 6001 scope.go:122] "RemoveContainer" containerID="26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b" Nov 03 17:35:38.455610 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:38.455579 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b\": container with ID starting with 26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b not found: ID does not exist" containerID="26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b" Nov 03 17:35:38.455722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.455618 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b"} err="failed to get container status \"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b\": rpc error: code = NotFound desc = could not find container \"26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b\": container with ID starting with 26a6ad619603cdfb908968fa9e2135aa55cad484a1193f0dc80d61b6f70a664b not found: ID does not exist" Nov 03 17:35:38.471336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.471277 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\" (UniqueName: \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\") pod \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " Nov 03 17:35:38.471502 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.471333 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\") pod \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\" (UID: \"7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389\") " Nov 03 17:35:38.471761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.471723 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume" pod "7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" (UID: "7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:38.473423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.473402 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9" pod "7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" (UID: "7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389"). InnerVolumeSpecName "kube-api-access-98zh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:38.572804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.572780 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\") pod \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " Nov 03 17:35:38.572970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.572855 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\" (UniqueName: \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\") pod \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\" (UID: \"c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842\") " Nov 03 17:35:38.573205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.573177 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume" pod "c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" (UID: "c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:38.573315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.573285 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98zh9\" (UniqueName: \"kubernetes.io/projected/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-kube-api-access-98zh9\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:38.573315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.573298 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:38.573315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.573307 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:38.574996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.574974 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd" pod "c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" (UID: "c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842"). InnerVolumeSpecName "kube-api-access-zvtfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:38.674724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.674695 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvtfd\" (UniqueName: \"kubernetes.io/projected/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842-kube-api-access-zvtfd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:38.716313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.716275 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg"] Nov 03 17:35:38.716996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.716972 6001 cpu_manager.go:414] "containerMap: removing container" podUID="319fc1ea-8e21-4421-9e32-b7e0d099b54c" containerName="revoke-keys" Nov 03 17:35:38.716996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.716996 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319fc1ea-8e21-4421-9e32-b7e0d099b54c" containerName="revoke-keys" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717033 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717041 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717078 6001 cpu_manager.go:414] "containerMap: removing container" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717085 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717096 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717103 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717131 6001 cpu_manager.go:414] "containerMap: removing container" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717138 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717170 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717179 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" Nov 03 17:35:38.717193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717196 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717204 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717360 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717376 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" containerName="payload-processing" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717387 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" containerName="payload-pre-processing" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717399 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="319fc1ea-8e21-4421-9e32-b7e0d099b54c" containerName="revoke-keys" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717420 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d52f73af-1985-484f-9c57-a4683e472589" containerName="payload-pre-processing" Nov 03 17:35:38.717805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.717432 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2617b18-379e-402d-9021-42c78c68d4be" containerName="payload-processing" Nov 03 17:35:38.722650 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.722629 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.738881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.738858 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:35:38.739463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.738957 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:35:38.743831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.743799 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg"] Nov 03 17:35:38.804504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.804473 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:35:38.808163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.808143 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e0e63d-96558b8c-9pgv4"] Nov 03 17:35:38.830964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.830933 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:35:38.831122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.831014 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:35:38.838480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.838440 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:35:38.845122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.845098 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e0e63d-6c6c8f9cf5-4vnq5"] Nov 03 17:35:38.876877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.876838 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwh6j\" (UniqueName: \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.877072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.877005 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.877072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.877053 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.978712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.978678 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwh6j\" (UniqueName: \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.978868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.978775 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.978868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.978825 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.979574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.979552 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:38.981169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:38.981140 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:39.025065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:39.025037 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwh6j\" (UniqueName: \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\") pod \"maas-api-revoke-sub-f4b998cb-nf8mg\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:39.036478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:39.036422 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:39.342535 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:35:39.342504 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded49247f_bf6d_45e8_bc86_cd474c1a50ed.slice/crio-6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51 WatchSource:0}: Error finding container 6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51: Status 404 returned error can't find the container with id 6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51 Nov 03 17:35:39.343158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:39.343136 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg"] Nov 03 17:35:39.441527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:39.441488 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" event={"ID":"ed49247f-bf6d-45e8-bc86-cd474c1a50ed","Type":"ContainerStarted","Data":"6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51"} Nov 03 17:35:40.256020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.255976 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:35:40.336525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.336418 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:35:40.340369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.340338 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:35:40.398182 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398150 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398238 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398322 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398391 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398425 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398435 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398521 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398624 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398674 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.398706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398685 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.399220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398707 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\") pod \"f988447e-72c6-4ea8-9b10-22d934337c34\" (UID: \"f988447e-72c6-4ea8-9b10-22d934337c34\") " Nov 03 17:35:40.399220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398775 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.399220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.398788 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:40.399220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399074 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.399484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399434 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f988447e-72c6-4ea8-9b10-22d934337c34-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.399484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399472 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.399611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399487 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.399611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399517 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.399611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.399532 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.401218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.401191 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.401349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.401319 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:40.401423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.401400 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.402006 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.401988 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p" pod "f988447e-72c6-4ea8-9b10-22d934337c34" (UID: "f988447e-72c6-4ea8-9b10-22d934337c34"). InnerVolumeSpecName "kube-api-access-wpv5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.453739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.453705 6001 generic.go:356] "Generic (PLEG): container finished" podID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerID="fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd" exitCode=0 Nov 03 17:35:40.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.453764 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" Nov 03 17:35:40.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.453801 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" event={"ID":"ea4becb0-ff63-43f0-a20c-b4bf9700d700","Type":"ContainerDied","Data":"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd"} Nov 03 17:35:40.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.453872 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc" event={"ID":"ea4becb0-ff63-43f0-a20c-b4bf9700d700","Type":"ContainerDied","Data":"dc2360099bc7c3e56416c5d3260bd24683a24a3ba97474156642ca5c86786664"} Nov 03 17:35:40.453920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.453904 6001 scope.go:122] "RemoveContainer" containerID="fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd" Nov 03 17:35:40.455846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.455824 6001 generic.go:356] "Generic (PLEG): container finished" podID="ed49247f-bf6d-45e8-bc86-cd474c1a50ed" containerID="dad93b033fac8c14e2478480ed026d8f21c967dea76fbe85384bd1ae09d4cde7" exitCode=0 Nov 03 17:35:40.455949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.455917 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" event={"ID":"ed49247f-bf6d-45e8-bc86-cd474c1a50ed","Type":"ContainerDied","Data":"dad93b033fac8c14e2478480ed026d8f21c967dea76fbe85384bd1ae09d4cde7"} Nov 03 17:35:40.457510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.457487 6001 generic.go:356] "Generic (PLEG): container finished" podID="1526dc29-4450-4968-8976-d965f11b1dd7" containerID="90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772" exitCode=0 Nov 03 17:35:40.457642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.457530 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" Nov 03 17:35:40.457727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.457698 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" event={"ID":"1526dc29-4450-4968-8976-d965f11b1dd7","Type":"ContainerDied","Data":"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772"} Nov 03 17:35:40.457793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.457748 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw" event={"ID":"1526dc29-4450-4968-8976-d965f11b1dd7","Type":"ContainerDied","Data":"2e486ce74a24fbd707e396ad380d9a926c676f2754c59885795f170f4557126e"} Nov 03 17:35:40.459205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.459187 6001 generic.go:356] "Generic (PLEG): container finished" podID="f988447e-72c6-4ea8-9b10-22d934337c34" containerID="6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29" exitCode=0 Nov 03 17:35:40.459308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.459235 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" Nov 03 17:35:40.459308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.459250 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" event={"ID":"f988447e-72c6-4ea8-9b10-22d934337c34","Type":"ContainerDied","Data":"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29"} Nov 03 17:35:40.459308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.459288 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl" event={"ID":"f988447e-72c6-4ea8-9b10-22d934337c34","Type":"ContainerDied","Data":"e3b7f52c064a9ecf6578c4250af67e198ff42d92ae19a9dda06347b02a2bbed5"} Nov 03 17:35:40.471782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.471766 6001 scope.go:122] "RemoveContainer" containerID="fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd" Nov 03 17:35:40.472079 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:40.472056 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd\": container with ID starting with fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd not found: ID does not exist" containerID="fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd" Nov 03 17:35:40.472172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.472090 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd"} err="failed to get container status \"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd\": rpc error: code = NotFound desc = could not find container \"fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd\": container with ID starting with fd0a986822b174169f7f88d8bf6da4d2922ba0e827bb613102ebdc4f1535b0dd not found: ID does not exist" Nov 03 17:35:40.472172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.472113 6001 scope.go:122] "RemoveContainer" containerID="90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772" Nov 03 17:35:40.482483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.482460 6001 scope.go:122] "RemoveContainer" containerID="90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772" Nov 03 17:35:40.482759 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:40.482738 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772\": container with ID starting with 90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772 not found: ID does not exist" containerID="90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772" Nov 03 17:35:40.482812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.482767 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772"} err="failed to get container status \"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772\": rpc error: code = NotFound desc = could not find container \"90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772\": container with ID starting with 90b7146c1c2ed59f8957132605edc4b70f5cc80f8876a685076e087065b3d772 not found: ID does not exist" Nov 03 17:35:40.482812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.482785 6001 scope.go:122] "RemoveContainer" containerID="6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29" Nov 03 17:35:40.492341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.492326 6001 scope.go:122] "RemoveContainer" containerID="6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29" Nov 03 17:35:40.492625 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:40.492605 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29\": container with ID starting with 6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29 not found: ID does not exist" containerID="6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29" Nov 03 17:35:40.492699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.492632 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29"} err="failed to get container status \"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29\": rpc error: code = NotFound desc = could not find container \"6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29\": container with ID starting with 6a5e68a02a47300c31837c79f422c4ab3ea590cb81ecb07c05c3e9dfad17ac29 not found: ID does not exist" Nov 03 17:35:40.500012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.499991 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.500086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500058 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.500130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500094 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.500167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500140 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.500207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500179 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500251 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500227 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500250 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.500289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500257 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.500402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500380 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500419 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500431 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.500608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500500 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500537 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.500920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500542 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:40.501014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500568 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.501014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500767 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:40.501014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500569 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.501014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501008 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500869 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501042 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500887 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.500957 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501148 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501193 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.501261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\") pod \"1526dc29-4450-4968-8976-d965f11b1dd7\" (UID: \"1526dc29-4450-4968-8976-d965f11b1dd7\") " Nov 03 17:35:40.501699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501272 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\") pod \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\" (UID: \"ea4becb0-ff63-43f0-a20c-b4bf9700d700\") " Nov 03 17:35:40.501699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501530 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.501823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.501729 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502045 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpv5p\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-kube-api-access-wpv5p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502073 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502095 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502109 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502121 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502136 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1526dc29-4450-4968-8976-d965f11b1dd7-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502151 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f988447e-72c6-4ea8-9b10-22d934337c34-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502169 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502182 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502194 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502205 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502219 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f988447e-72c6-4ea8-9b10-22d934337c34-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502231 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f988447e-72c6-4ea8-9b10-22d934337c34-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.502269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.502244 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.503418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.503335 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.503418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.503367 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "kube-api-access-q2lpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.503600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.503418 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.503600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.503521 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "kube-api-access-r6488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.503863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.503845 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:40.504130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.504114 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:40.504528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.504508 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo" pod "ea4becb0-ff63-43f0-a20c-b4bf9700d700" (UID: "ea4becb0-ff63-43f0-a20c-b4bf9700d700"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:40.504864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.504843 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo" pod "1526dc29-4450-4968-8976-d965f11b1dd7" (UID: "1526dc29-4450-4968-8976-d965f11b1dd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:35:40.595349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.595259 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:35:40.601716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.601684 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-jmwhl"] Nov 03 17:35:40.603523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603503 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603528 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2lpq\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-kube-api-access-q2lpq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603539 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603549 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1526dc29-4450-4968-8976-d965f11b1dd7-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603557 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4becb0-ff63-43f0-a20c-b4bf9700d700-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603565 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1526dc29-4450-4968-8976-d965f11b1dd7-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603573 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1526dc29-4450-4968-8976-d965f11b1dd7-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.603773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.603582 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6488\" (UniqueName: \"kubernetes.io/projected/ea4becb0-ff63-43f0-a20c-b4bf9700d700-kube-api-access-r6488\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:40.789544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.789511 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389" path="/var/lib/kubelet/pods/7f2f9ab3-b8a9-4d82-8bcb-66a1a284b389/volumes" Nov 03 17:35:40.790000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.789985 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842" path="/var/lib/kubelet/pods/c7aa3bd7-b7fb-46e8-897e-1fd99f4f0842/volumes" Nov 03 17:35:40.790399 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.790386 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" path="/var/lib/kubelet/pods/f988447e-72c6-4ea8-9b10-22d934337c34/volumes" Nov 03 17:35:40.803286 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.803262 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:35:40.807969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.807949 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-t64tc"] Nov 03 17:35:40.866517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.866425 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:35:40.869786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:40.869760 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-2h7hw"] Nov 03 17:35:41.626964 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.626933 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:41.817463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.817420 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\") pod \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " Nov 03 17:35:41.817698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.817679 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\") pod \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " Nov 03 17:35:41.817796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.817742 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\" (UniqueName: \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\") pod \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\" (UID: \"ed49247f-bf6d-45e8-bc86-cd474c1a50ed\") " Nov 03 17:35:41.818039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.818012 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca" pod "ed49247f-bf6d-45e8-bc86-cd474c1a50ed" (UID: "ed49247f-bf6d-45e8-bc86-cd474c1a50ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:41.818181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.818167 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.819651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.819630 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca" pod "ed49247f-bf6d-45e8-bc86-cd474c1a50ed" (UID: "ed49247f-bf6d-45e8-bc86-cd474c1a50ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:41.819805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.819785 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j" pod "ed49247f-bf6d-45e8-bc86-cd474c1a50ed" (UID: "ed49247f-bf6d-45e8-bc86-cd474c1a50ed"). InnerVolumeSpecName "kube-api-access-dwh6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:41.919426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.919387 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwh6j\" (UniqueName: \"kubernetes.io/projected/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-kube-api-access-dwh6j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:41.919426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:41.919426 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed49247f-bf6d-45e8-bc86-cd474c1a50ed-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:42.477713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:42.477677 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" Nov 03 17:35:42.477713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:42.477705 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg" event={"ID":"ed49247f-bf6d-45e8-bc86-cd474c1a50ed","Type":"ContainerDied","Data":"6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51"} Nov 03 17:35:42.477953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:42.477782 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f8d153f504fe016ffd0de8aa1003f5d316b72f5336e3dbe1d4c44b1a19dba51" Nov 03 17:35:42.789645 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:42.789554 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" path="/var/lib/kubelet/pods/1526dc29-4450-4968-8976-d965f11b1dd7/volumes" Nov 03 17:35:42.790259 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:42.790237 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" path="/var/lib/kubelet/pods/ea4becb0-ff63-43f0-a20c-b4bf9700d700/volumes" Nov 03 17:35:44.512808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:44.512769 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:44.513252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:44.512773 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:44.561049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:44.561011 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 17:35:49.328135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328102 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b"] Nov 03 17:35:49.328692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328675 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" Nov 03 17:35:49.328736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328693 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" Nov 03 17:35:49.328736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328718 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ed49247f-bf6d-45e8-bc86-cd474c1a50ed" containerName="revoke-keys" Nov 03 17:35:49.328736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328723 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed49247f-bf6d-45e8-bc86-cd474c1a50ed" containerName="revoke-keys" Nov 03 17:35:49.328846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328741 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" Nov 03 17:35:49.328846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328749 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" Nov 03 17:35:49.328846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328764 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" Nov 03 17:35:49.328846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328769 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" Nov 03 17:35:49.329013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328855 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1526dc29-4450-4968-8976-d965f11b1dd7" containerName="istio-proxy" Nov 03 17:35:49.329013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328864 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4becb0-ff63-43f0-a20c-b4bf9700d700" containerName="istio-proxy" Nov 03 17:35:49.329013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328876 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ed6a75-097f-471c-92f9-2356c11edb19" containerName="istio-proxy" Nov 03 17:35:49.329013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328884 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f988447e-72c6-4ea8-9b10-22d934337c34" containerName="istio-proxy" Nov 03 17:35:49.329013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.328892 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed49247f-bf6d-45e8-bc86-cd474c1a50ed" containerName="revoke-keys" Nov 03 17:35:49.332462 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.332422 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.352965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.352932 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b"] Nov 03 17:35:49.361977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.361946 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:35:49.361977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.361956 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:35:49.362275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.362014 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:35:49.362275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.362014 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:35:49.405545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.405510 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.405700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.405673 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbkd\" (UniqueName: \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.405740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.405714 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.506986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.506953 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.507213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.507065 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbkd\" (UniqueName: \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.507213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.507097 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.507853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.507831 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.509400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.509376 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.530679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.530644 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbkd\" (UniqueName: \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\") pod \"maas-api-revoke-sub-5ec88cbb-59w5b\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.561305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.561262 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" containerID="cri-o://7c013c08d1d6e517edd6db24723ca087fe1b9008ce7105388289bc574fd07ac5" gracePeriod=25 Nov 03 17:35:49.646380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.646293 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:49.952923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:49.952892 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b"] Nov 03 17:35:50.534926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:50.534828 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" event={"ID":"304852bb-c286-4b2f-b407-3eb5516072b6","Type":"ContainerStarted","Data":"a49f1c42a45304d58ac0863e5fa2f957df24738128ca8db4eb8835d4b11b7bbc"} Nov 03 17:35:50.534926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:50.534885 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" event={"ID":"304852bb-c286-4b2f-b407-3eb5516072b6","Type":"ContainerStarted","Data":"ecc1ac53c9dac60d373ac17a45f1d61147fc9c996ef4e9009175669e6ab1b2fa"} Nov 03 17:35:50.581009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:50.580946 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" podStartSLOduration=1.580931336 podStartE2EDuration="1.580931336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:35:50.577914381 +0000 UTC m=+4845.911127966" watchObservedRunningTime="2026-11-03 17:35:50.580931336 +0000 UTC m=+4845.914144908" Nov 03 17:35:51.545239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:51.545197 6001 generic.go:356] "Generic (PLEG): container finished" podID="304852bb-c286-4b2f-b407-3eb5516072b6" containerID="a49f1c42a45304d58ac0863e5fa2f957df24738128ca8db4eb8835d4b11b7bbc" exitCode=0 Nov 03 17:35:51.545669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:51.545281 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" event={"ID":"304852bb-c286-4b2f-b407-3eb5516072b6","Type":"ContainerDied","Data":"a49f1c42a45304d58ac0863e5fa2f957df24738128ca8db4eb8835d4b11b7bbc"} Nov 03 17:35:52.252294 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:52.252259 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:35:52.706218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.706188 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:52.762406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.762368 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\") pod \"304852bb-c286-4b2f-b407-3eb5516072b6\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " Nov 03 17:35:52.762647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.762628 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\" (UniqueName: \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\") pod \"304852bb-c286-4b2f-b407-3eb5516072b6\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " Nov 03 17:35:52.762732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.762686 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\") pod \"304852bb-c286-4b2f-b407-3eb5516072b6\" (UID: \"304852bb-c286-4b2f-b407-3eb5516072b6\") " Nov 03 17:35:52.763484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.763424 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca" pod "304852bb-c286-4b2f-b407-3eb5516072b6" (UID: "304852bb-c286-4b2f-b407-3eb5516072b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:52.765334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.765303 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca" pod "304852bb-c286-4b2f-b407-3eb5516072b6" (UID: "304852bb-c286-4b2f-b407-3eb5516072b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:52.765705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.765678 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd" pod "304852bb-c286-4b2f-b407-3eb5516072b6" (UID: "304852bb-c286-4b2f-b407-3eb5516072b6"). InnerVolumeSpecName "kube-api-access-tgbkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:52.864136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.864036 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:52.864136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.864065 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304852bb-c286-4b2f-b407-3eb5516072b6-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:52.864136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:52.864079 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgbkd\" (UniqueName: \"kubernetes.io/projected/304852bb-c286-4b2f-b407-3eb5516072b6-kube-api-access-tgbkd\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:53.423346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:53.423312 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:53.561540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:53.561503 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" event={"ID":"304852bb-c286-4b2f-b407-3eb5516072b6","Type":"ContainerDied","Data":"ecc1ac53c9dac60d373ac17a45f1d61147fc9c996ef4e9009175669e6ab1b2fa"} Nov 03 17:35:53.561749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:53.561555 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc1ac53c9dac60d373ac17a45f1d61147fc9c996ef4e9009175669e6ab1b2fa" Nov 03 17:35:53.561749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:53.561517 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b" Nov 03 17:35:54.511968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:54.511930 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:54.511968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:54.511951 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:54.722160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:54.722124 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 17:35:54.722394 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:54.722366 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" containerID="cri-o://9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39" gracePeriod=30 Nov 03 17:35:56.034730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:56.034665 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 17:35:57.380225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.380196 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:35:57.413559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.413527 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:35:57.420778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.420751 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\") pod \"4bef957b-baa4-4381-a556-cfa92982761f\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " Nov 03 17:35:57.421024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.420998 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\" (UniqueName: \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\") pod \"4bef957b-baa4-4381-a556-cfa92982761f\" (UID: \"4bef957b-baa4-4381-a556-cfa92982761f\") " Nov 03 17:35:57.421646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.421388 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume" pod "4bef957b-baa4-4381-a556-cfa92982761f" (UID: "4bef957b-baa4-4381-a556-cfa92982761f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:57.421646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.421616 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bef957b-baa4-4381-a556-cfa92982761f-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:57.423893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.423864 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d" pod "4bef957b-baa4-4381-a556-cfa92982761f" (UID: "4bef957b-baa4-4381-a556-cfa92982761f"). InnerVolumeSpecName "kube-api-access-ctd9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:57.522675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.522645 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\" (UniqueName: \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\") pod \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " Nov 03 17:35:57.522873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.522779 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\") pod \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\" (UID: \"a38d94fd-440f-41d7-9a8b-1a3b42fc378a\") " Nov 03 17:35:57.523139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.523124 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctd9d\" (UniqueName: \"kubernetes.io/projected/4bef957b-baa4-4381-a556-cfa92982761f-kube-api-access-ctd9d\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:57.523231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.523132 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume" pod "a38d94fd-440f-41d7-9a8b-1a3b42fc378a" (UID: "a38d94fd-440f-41d7-9a8b-1a3b42fc378a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:57.524796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.524775 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv" pod "a38d94fd-440f-41d7-9a8b-1a3b42fc378a" (UID: "a38d94fd-440f-41d7-9a8b-1a3b42fc378a"). InnerVolumeSpecName "kube-api-access-k4scv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:57.591956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.591863 6001 generic.go:356] "Generic (PLEG): container finished" podID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerID="7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e" exitCode=0 Nov 03 17:35:57.591956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.591909 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" Nov 03 17:35:57.592189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.591949 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" event={"ID":"a38d94fd-440f-41d7-9a8b-1a3b42fc378a","Type":"ContainerDied","Data":"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e"} Nov 03 17:35:57.592189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.592000 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7" event={"ID":"a38d94fd-440f-41d7-9a8b-1a3b42fc378a","Type":"ContainerDied","Data":"bfec078242c73e38d5fdc77453234034a349547c2cd43528c6b5e17b21f5499e"} Nov 03 17:35:57.592189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.592022 6001 scope.go:122] "RemoveContainer" containerID="7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e" Nov 03 17:35:57.593669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.593650 6001 generic.go:356] "Generic (PLEG): container finished" podID="4bef957b-baa4-4381-a556-cfa92982761f" containerID="7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d" exitCode=0 Nov 03 17:35:57.593776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.593696 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" Nov 03 17:35:57.593776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.593709 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" event={"ID":"4bef957b-baa4-4381-a556-cfa92982761f","Type":"ContainerDied","Data":"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d"} Nov 03 17:35:57.593776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.593748 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv" event={"ID":"4bef957b-baa4-4381-a556-cfa92982761f","Type":"ContainerDied","Data":"a852b7e6731fe4617550fffc3b8d363d987f043be3bcd2ac1d4bebaaa5a9547f"} Nov 03 17:35:57.610970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.610948 6001 scope.go:122] "RemoveContainer" containerID="7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e" Nov 03 17:35:57.611498 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:57.611462 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e\": container with ID starting with 7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e not found: ID does not exist" containerID="7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e" Nov 03 17:35:57.611629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.611513 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e"} err="failed to get container status \"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e\": rpc error: code = NotFound desc = could not find container \"7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e\": container with ID starting with 7936f1dd1afaa5188fd961d67fc2cf597d341947d8f4ff61aab719c1331d2b3e not found: ID does not exist" Nov 03 17:35:57.611629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.611540 6001 scope.go:122] "RemoveContainer" containerID="7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d" Nov 03 17:35:57.625903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.625874 6001 scope.go:122] "RemoveContainer" containerID="7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d" Nov 03 17:35:57.626470 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:35:57.626414 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d\": container with ID starting with 7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d not found: ID does not exist" containerID="7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d" Nov 03 17:35:57.626583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.626484 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d"} err="failed to get container status \"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d\": rpc error: code = NotFound desc = could not find container \"7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d\": container with ID starting with 7c28ca153279c583e2b212b7f95b1e7294358d4ca9d30da235c2a6efa9abfd1d not found: ID does not exist" Nov 03 17:35:57.626715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.626666 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4scv\" (UniqueName: \"kubernetes.io/projected/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-kube-api-access-k4scv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:57.626715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.626695 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a38d94fd-440f-41d7-9a8b-1a3b42fc378a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:35:57.656252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.656216 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:35:57.673790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.673753 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ca9d7c-7d4775b7b7-w82v7"] Nov 03 17:35:57.724503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.724468 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:35:57.734220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.734187 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-ca9d7c-5dcb467b7-967qv"] Nov 03 17:35:57.743266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.743230 6001 patch_prober.go:29] interesting pod/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:35:57.743431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:57.743294 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:35:58.789882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:58.789849 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bef957b-baa4-4381-a556-cfa92982761f" path="/var/lib/kubelet/pods/4bef957b-baa4-4381-a556-cfa92982761f/volumes" Nov 03 17:35:58.790301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:58.790281 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" path="/var/lib/kubelet/pods/a38d94fd-440f-41d7-9a8b-1a3b42fc378a/volumes" Nov 03 17:35:59.995343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:35:59.995284 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 17:36:00.054599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054568 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.054819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054650 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.054819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054677 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.054819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054709 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.054819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054735 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.054819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054808 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.055150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054884 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.055150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054894 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:00.055150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.054990 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.055150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055034 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\") pod \"89b830bc-c904-47e8-a5a3-f33cfaafff32\" (UID: \"89b830bc-c904-47e8-a5a3-f33cfaafff32\") " Nov 03 17:36:00.055398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055030 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:00.055398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055349 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:00.055551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055402 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:00.055686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055663 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.055759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055688 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.055759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055703 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.055759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055717 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.055941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.055877 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:00.057313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.057287 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:00.057313 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.057302 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "kube-api-access-nrv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:00.057577 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.057416 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:00.057577 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.057551 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token" pod "89b830bc-c904-47e8-a5a3-f33cfaafff32" (UID: "89b830bc-c904-47e8-a5a3-f33cfaafff32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:00.156856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.156817 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.157042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.156862 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.157042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.156878 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b830bc-c904-47e8-a5a3-f33cfaafff32-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.157042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.156892 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.157042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.156905 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrv4w\" (UniqueName: \"kubernetes.io/projected/89b830bc-c904-47e8-a5a3-f33cfaafff32-kube-api-access-nrv4w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:00.619142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.619109 6001 generic.go:356] "Generic (PLEG): container finished" podID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerID="9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39" exitCode=0 Nov 03 17:36:00.619354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.619176 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" Nov 03 17:36:00.619354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.619185 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerDied","Data":"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39"} Nov 03 17:36:00.619354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.619238 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz" event={"ID":"89b830bc-c904-47e8-a5a3-f33cfaafff32","Type":"ContainerDied","Data":"5d86678de4de5940ea5f5c04b9092856cb2b9a322e30d9b4103f875b05c93220"} Nov 03 17:36:00.619354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.619263 6001 scope.go:122] "RemoveContainer" containerID="9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39" Nov 03 17:36:00.632855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.632836 6001 scope.go:122] "RemoveContainer" containerID="f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a" Nov 03 17:36:00.669604 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.669572 6001 scope.go:122] "RemoveContainer" containerID="9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39" Nov 03 17:36:00.669993 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:00.669969 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39\": container with ID starting with 9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39 not found: ID does not exist" containerID="9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39" Nov 03 17:36:00.670094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.670001 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39"} err="failed to get container status \"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39\": rpc error: code = NotFound desc = could not find container \"9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39\": container with ID starting with 9ec887f2ae9f01519784422e8a0ef17f71f62b474dda2e3fc736a33a44de7e39 not found: ID does not exist" Nov 03 17:36:00.670094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.670023 6001 scope.go:122] "RemoveContainer" containerID="f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a" Nov 03 17:36:00.670304 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:00.670287 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a\": container with ID starting with f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a not found: ID does not exist" containerID="f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a" Nov 03 17:36:00.670372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.670307 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a"} err="failed to get container status \"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a\": rpc error: code = NotFound desc = could not find container \"f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a\": container with ID starting with f719fbda28885965ba93edeb79e4ece60a521c0a38bc04d30579208bdf3d168a not found: ID does not exist" Nov 03 17:36:00.695895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.695861 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 17:36:00.731763 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.731732 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-5a5ab1-openshift-default-cbb74d5b9-hlrlz"] Nov 03 17:36:00.789673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:00.789633 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" path="/var/lib/kubelet/pods/89b830bc-c904-47e8-a5a3-f33cfaafff32/volumes" Nov 03 17:36:01.036058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:01.036005 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" containerID="cri-o://1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974" gracePeriod=25 Nov 03 17:36:02.916682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:02.916640 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 17:36:03.423223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:03.423184 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:07.917610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:07.917559 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" containerID="cri-o://df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb" gracePeriod=25 Nov 03 17:36:08.030815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:08.030779 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh"] Nov 03 17:36:08.043303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:08.043262 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef080186-bqmrh"] Nov 03 17:36:08.790728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:08.790693 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a00b48f-d590-45e4-b9d3-5d17e48bb9cc" path="/var/lib/kubelet/pods/0a00b48f-d590-45e4-b9d3-5d17e48bb9cc/volumes" Nov 03 17:36:09.460177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.460130 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:09.688541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.688509 6001 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerID="7c013c08d1d6e517edd6db24723ca087fe1b9008ce7105388289bc574fd07ac5" exitCode=0 Nov 03 17:36:09.688706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.688584 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" event={"ID":"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4","Type":"ContainerDied","Data":"7c013c08d1d6e517edd6db24723ca087fe1b9008ce7105388289bc574fd07ac5"} Nov 03 17:36:09.825425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.825401 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 17:36:09.993521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.993417 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\") pod \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " Nov 03 17:36:09.993789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.993768 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\" (UniqueName: \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\") pod \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\" (UID: \"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4\") " Nov 03 17:36:09.993884 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.993799 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume" pod "1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" (UID: "1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:09.994143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.994125 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:09.995988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:09.995961 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68" pod "1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" (UID: "1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4"). InnerVolumeSpecName "kube-api-access-z2f68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:10.094907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.094876 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2f68\" (UniqueName: \"kubernetes.io/projected/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4-kube-api-access-z2f68\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.696503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.696470 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" event={"ID":"1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4","Type":"ContainerDied","Data":"0cbdb9d07637b34014fa587e6681edaab70e8176271aa94f770cbad3ded19010"} Nov 03 17:36:10.696503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.696500 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc" Nov 03 17:36:10.697011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.696537 6001 scope.go:122] "RemoveContainer" containerID="7c013c08d1d6e517edd6db24723ca087fe1b9008ce7105388289bc574fd07ac5" Nov 03 17:36:10.755898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.755814 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 17:36:10.767352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.767291 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5a5ab1-6499bc975f-qg5rc"] Nov 03 17:36:10.791049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:10.791008 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" path="/var/lib/kubelet/pods/1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4/volumes" Nov 03 17:36:13.831535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:13.831505 6001 scope.go:122] "RemoveContainer" containerID="89a922f5745ceaa0e9d1abbec147d04a0f8cf0efa0ab8331e1323d924379ae83" Nov 03 17:36:14.215761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:14.215667 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:17.979534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.979409 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85"] Nov 03 17:36:17.980176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980150 6001 cpu_manager.go:414] "containerMap: removing container" podUID="304852bb-c286-4b2f-b407-3eb5516072b6" containerName="revoke-keys" Nov 03 17:36:17.980280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980177 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304852bb-c286-4b2f-b407-3eb5516072b6" containerName="revoke-keys" Nov 03 17:36:17.980280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980227 6001 cpu_manager.go:414] "containerMap: removing container" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:17.980280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980237 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:17.980280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980248 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" Nov 03 17:36:17.980280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980256 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980288 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980296 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980323 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980331 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980488 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bef957b-baa4-4381-a556-cfa92982761f" containerName="payload-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980506 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="304852bb-c286-4b2f-b407-3eb5516072b6" containerName="revoke-keys" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980524 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980533 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d8d8b98-9f22-4f9e-9bc1-871e9d02bfc4" containerName="payload-processing" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980546 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:17.980606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.980556 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a38d94fd-440f-41d7-9a8b-1a3b42fc378a" containerName="payload-pre-processing" Nov 03 17:36:17.986003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:17.985980 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.004480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.004437 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85"] Nov 03 17:36:18.013789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.013763 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:36:18.013789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.013764 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:36:18.013993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.013820 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:36:18.013993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.013884 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:36:18.095574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.095539 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.095753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.095659 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.095799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.095776 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2sl4\" (UniqueName: \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.196794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.196742 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.197030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.196845 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.197030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.196965 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2sl4\" (UniqueName: \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.197589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.197563 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.199309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.199284 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.221522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.221491 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2sl4\" (UniqueName: \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\") pod \"maas-api-revoke-sub-84906a43-htz85\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.299747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.299658 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:18.604066 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:36:18.604034 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ebc72a_748b_4fbe_8e2e_b11a4b766ef5.slice/crio-90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34 WatchSource:0}: Error finding container 90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34: Status 404 returned error can't find the container with id 90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34 Nov 03 17:36:18.605926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.605891 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85"] Nov 03 17:36:18.752017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:18.751983 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" event={"ID":"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5","Type":"ContainerStarted","Data":"90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34"} Nov 03 17:36:19.459933 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:19.459897 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:19.760790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:19.760751 6001 generic.go:356] "Generic (PLEG): container finished" podID="b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" containerID="93c4b61bdbc3272ced84f3992df48242ef8e414644b167f8e6e0aff81befa981" exitCode=0 Nov 03 17:36:19.761004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:19.760836 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" event={"ID":"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5","Type":"ContainerDied","Data":"93c4b61bdbc3272ced84f3992df48242ef8e414644b167f8e6e0aff81befa981"} Nov 03 17:36:20.551140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:20.551104 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 17:36:20.927361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:20.927332 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:21.030279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.030246 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\") pod \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " Nov 03 17:36:21.030528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.030348 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\" (UniqueName: \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\") pod \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " Nov 03 17:36:21.030528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.030434 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\") pod \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\" (UID: \"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5\") " Nov 03 17:36:21.030695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.030661 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca" pod "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" (UID: "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:21.030845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.030827 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.032658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.032622 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca" pod "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" (UID: "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:21.032762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.032661 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4" pod "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" (UID: "b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5"). InnerVolumeSpecName "kube-api-access-r2sl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:21.132702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.132556 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2sl4\" (UniqueName: \"kubernetes.io/projected/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-kube-api-access-r2sl4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.132702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.132662 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.285134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.285108 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 17:36:21.334433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.334394 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\") pod \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " Nov 03 17:36:21.334669 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.334481 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\" (UniqueName: \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\") pod \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\" (UID: \"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2\") " Nov 03 17:36:21.334853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.334829 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume" pod "bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" (UID: "bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:21.337250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.337215 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv" pod "bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" (UID: "bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2"). InnerVolumeSpecName "kube-api-access-t8cfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:21.435842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.435745 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.435842 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.435781 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8cfv\" (UniqueName: \"kubernetes.io/projected/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2-kube-api-access-t8cfv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.777982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.777950 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" Nov 03 17:36:21.777982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.777970 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85" event={"ID":"b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5","Type":"ContainerDied","Data":"90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34"} Nov 03 17:36:21.778629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.778035 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b5cdfe0ea90352b224b297f01bee4d264ad7f4ba26beccf1fff9a175cc9b34" Nov 03 17:36:21.779514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.779489 6001 generic.go:356] "Generic (PLEG): container finished" podID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerID="1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974" exitCode=0 Nov 03 17:36:21.779638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.779519 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" event={"ID":"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2","Type":"ContainerDied","Data":"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974"} Nov 03 17:36:21.779638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.779543 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" Nov 03 17:36:21.779638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.779577 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq" event={"ID":"bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2","Type":"ContainerDied","Data":"b20c022683131a12cc373bff684460d65caf8afee5c6c6a141e8f8f6b197c375"} Nov 03 17:36:21.779638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.779611 6001 scope.go:122] "RemoveContainer" containerID="1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974" Nov 03 17:36:21.791630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.791542 6001 scope.go:122] "RemoveContainer" containerID="1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974" Nov 03 17:36:21.791975 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:21.791949 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974\": container with ID starting with 1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974 not found: ID does not exist" containerID="1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974" Nov 03 17:36:21.792084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.791983 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974"} err="failed to get container status \"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974\": rpc error: code = NotFound desc = could not find container \"1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974\": container with ID starting with 1b1ce95b597773ae447d830e01fcd3f28fc0cebeb03b9bc74fabd6f3f7f56974 not found: ID does not exist" Nov 03 17:36:21.836405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.836366 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 17:36:21.848851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:21.848818 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3b6f35-76cd4b4875-kt2rq"] Nov 03 17:36:22.251598 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:22.251561 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:36:22.567821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:22.567713 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 17:36:22.788967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:22.788931 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" path="/var/lib/kubelet/pods/bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2/volumes" Nov 03 17:36:24.216002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:24.215955 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:25.552047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:25.551991 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" containerID="cri-o://bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783" gracePeriod=25 Nov 03 17:36:27.034298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:27.034255 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv"] Nov 03 17:36:27.046719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:27.046685 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6cb243a-7p9sv"] Nov 03 17:36:28.177405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.177327 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 17:36:28.324777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.324741 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\") pod \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " Nov 03 17:36:28.324985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.324857 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\" (UniqueName: \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\") pod \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\" (UID: \"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a\") " Nov 03 17:36:28.325164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.325139 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume" pod "aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" (UID: "aa90afb0-a1e4-45ee-bf57-b8488ea55f7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:28.325432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.325414 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.327058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.327035 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq" pod "aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" (UID: "aa90afb0-a1e4-45ee-bf57-b8488ea55f7a"). InnerVolumeSpecName "kube-api-access-xzzcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:28.426411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.426365 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzcq\" (UniqueName: \"kubernetes.io/projected/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a-kube-api-access-xzzcq\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.525883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.525836 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 17:36:28.556701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.556664 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 17:36:28.789664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.789585 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfbd6dab-544b-4a88-91d5-8cb1e7018b15" path="/var/lib/kubelet/pods/cfbd6dab-544b-4a88-91d5-8cb1e7018b15/volumes" Nov 03 17:36:28.829924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.829885 6001 generic.go:356] "Generic (PLEG): container finished" podID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerID="df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb" exitCode=0 Nov 03 17:36:28.830143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.829956 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" Nov 03 17:36:28.830143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.829965 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" event={"ID":"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a","Type":"ContainerDied","Data":"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb"} Nov 03 17:36:28.830143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.830015 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq" event={"ID":"aa90afb0-a1e4-45ee-bf57-b8488ea55f7a","Type":"ContainerDied","Data":"edf052385f7287598e0d0226471acf112671069f08b338e3e780ef41059d2407"} Nov 03 17:36:28.830143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.830038 6001 scope.go:122] "RemoveContainer" containerID="df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb" Nov 03 17:36:28.842044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.842022 6001 scope.go:122] "RemoveContainer" containerID="df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb" Nov 03 17:36:28.842419 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:28.842396 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb\": container with ID starting with df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb not found: ID does not exist" containerID="df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb" Nov 03 17:36:28.842514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.842430 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb"} err="failed to get container status \"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb\": rpc error: code = NotFound desc = could not find container \"df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb\": container with ID starting with df30ca282cf6333fc80b52eaac0b3945bc7ead5ff35be5d8811b6d09f9c52beb not found: ID does not exist" Nov 03 17:36:28.863092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863059 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7"] Nov 03 17:36:28.863654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863636 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" containerName="revoke-keys" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863656 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" containerName="revoke-keys" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863676 6001 cpu_manager.go:414] "containerMap: removing container" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863682 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863691 6001 cpu_manager.go:414] "containerMap: removing container" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863695 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b830bc-c904-47e8-a5a3-f33cfaafff32" containerName="istio-proxy" Nov 03 17:36:28.863706 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863706 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" Nov 03 17:36:28.863950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863711 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" Nov 03 17:36:28.863950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863805 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" containerName="revoke-keys" Nov 03 17:36:28.863950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863815 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" containerName="payload-processing" Nov 03 17:36:28.863950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.863822 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdc4bde-4c9c-4ae4-8953-da05fb0a6eb2" containerName="payload-pre-processing" Nov 03 17:36:28.868985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.868958 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:28.890225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.888217 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 17:36:28.890225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.888827 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:36:28.890225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.888886 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:36:28.890225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.888946 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:36:28.890225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.888958 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:36:28.891995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.891970 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7"] Nov 03 17:36:28.895743 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:28.895720 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-1e9d48-75fd7cb4f7-msgkq"] Nov 03 17:36:29.032095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.032049 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.032305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.032254 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgk7t\" (UniqueName: \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.032305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.032299 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.133722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.133614 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgk7t\" (UniqueName: \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.133722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.133663 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.133962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.133751 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.134652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.134625 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.136746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.136720 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.178785 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.178753 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgk7t\" (UniqueName: \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\") pod \"maas-api-revoke-sub-3dd760d1-l5wd7\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.183580 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.183555 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:29.493227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.491711 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7"] Nov 03 17:36:29.838602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.838564 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" event={"ID":"ad26f876-121c-4472-94d9-5e13411515cf","Type":"ContainerStarted","Data":"e085e2c51dd73cadbbe6a350b0a278f80cfa03cd334cbc7d824c05d3fd86ae70"} Nov 03 17:36:29.838822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.838631 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" event={"ID":"ad26f876-121c-4472-94d9-5e13411515cf","Type":"ContainerStarted","Data":"b6aaad712c2d15a04f4acd7df81441be1aa85bdaa6d5c8a91bcc566253949011"} Nov 03 17:36:29.883864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:29.883796 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" podStartSLOduration=1.883782372 podStartE2EDuration="1.883782372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:29.881713905 +0000 UTC m=+4885.214927474" watchObservedRunningTime="2026-11-03 17:36:29.883782372 +0000 UTC m=+4885.216995941" Nov 03 17:36:30.789134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:30.789100 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa90afb0-a1e4-45ee-bf57-b8488ea55f7a" path="/var/lib/kubelet/pods/aa90afb0-a1e4-45ee-bf57-b8488ea55f7a/volumes" Nov 03 17:36:30.848152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:30.848119 6001 generic.go:356] "Generic (PLEG): container finished" podID="ad26f876-121c-4472-94d9-5e13411515cf" containerID="e085e2c51dd73cadbbe6a350b0a278f80cfa03cd334cbc7d824c05d3fd86ae70" exitCode=0 Nov 03 17:36:30.848363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:30.848212 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" event={"ID":"ad26f876-121c-4472-94d9-5e13411515cf","Type":"ContainerDied","Data":"e085e2c51dd73cadbbe6a350b0a278f80cfa03cd334cbc7d824c05d3fd86ae70"} Nov 03 17:36:32.012558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.012528 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:32.171859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.171749 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\") pod \"ad26f876-121c-4472-94d9-5e13411515cf\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " Nov 03 17:36:32.171859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.171817 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\") pod \"ad26f876-121c-4472-94d9-5e13411515cf\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " Nov 03 17:36:32.172134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.172019 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\" (UniqueName: \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\") pod \"ad26f876-121c-4472-94d9-5e13411515cf\" (UID: \"ad26f876-121c-4472-94d9-5e13411515cf\") " Nov 03 17:36:32.172245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.172214 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca" pod "ad26f876-121c-4472-94d9-5e13411515cf" (UID: "ad26f876-121c-4472-94d9-5e13411515cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:32.172417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.172394 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:32.174135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.174108 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca" pod "ad26f876-121c-4472-94d9-5e13411515cf" (UID: "ad26f876-121c-4472-94d9-5e13411515cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:32.174135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.174122 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t" pod "ad26f876-121c-4472-94d9-5e13411515cf" (UID: "ad26f876-121c-4472-94d9-5e13411515cf"). InnerVolumeSpecName "kube-api-access-lgk7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:32.273274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.273224 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad26f876-121c-4472-94d9-5e13411515cf-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:32.273274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.273264 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgk7t\" (UniqueName: \"kubernetes.io/projected/ad26f876-121c-4472-94d9-5e13411515cf-kube-api-access-lgk7t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:32.873314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.873283 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" Nov 03 17:36:32.873521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.873284 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7" event={"ID":"ad26f876-121c-4472-94d9-5e13411515cf","Type":"ContainerDied","Data":"b6aaad712c2d15a04f4acd7df81441be1aa85bdaa6d5c8a91bcc566253949011"} Nov 03 17:36:32.873521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:32.873417 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6aaad712c2d15a04f4acd7df81441be1aa85bdaa6d5c8a91bcc566253949011" Nov 03 17:36:33.526504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:33.526420 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" containerID="cri-o://eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e" gracePeriod=25 Nov 03 17:36:33.556991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:33.556921 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" containerID="cri-o://3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f" gracePeriod=25 Nov 03 17:36:34.865378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:34.865340 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:35.744564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:35.744525 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:36:35.744817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:35.744792 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" containerID="cri-o://d837c3bce145c6d7556f06d15bc8449db9a9599a343fec16a6d52c65bccba89b" gracePeriod=30 Nov 03 17:36:36.561838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:36.561794 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 17:36:36.562375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:36.562015 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" containerID="cri-o://5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b" gracePeriod=30 Nov 03 17:36:37.547547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.547509 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx"] Nov 03 17:36:37.548063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.548048 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ad26f876-121c-4472-94d9-5e13411515cf" containerName="revoke-keys" Nov 03 17:36:37.548138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.548063 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad26f876-121c-4472-94d9-5e13411515cf" containerName="revoke-keys" Nov 03 17:36:37.548191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.548164 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad26f876-121c-4472-94d9-5e13411515cf" containerName="revoke-keys" Nov 03 17:36:37.551726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.551702 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.572799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.572764 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx"] Nov 03 17:36:37.581057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.581005 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:36:37.581262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.581059 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:36:37.581262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.581063 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:36:37.581262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.581188 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:36:37.589657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.589623 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" containerID="cri-o://611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11" gracePeriod=45 Nov 03 17:36:37.641973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.641930 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74h5r\" (UniqueName: \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.642172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.642073 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.642234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.642184 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.742910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.742870 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.743104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.742985 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.743104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.743061 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74h5r\" (UniqueName: \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.743648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.743626 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.745662 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.745634 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.769686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.768527 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74h5r\" (UniqueName: \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\") pod \"maas-api-revoke-sub-0213989f-klxmx\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.865143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.865114 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:37.868357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.868334 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:36:37.906804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.906763 6001 generic.go:356] "Generic (PLEG): container finished" podID="59800126-e27d-4638-879e-77c88cc5a506" containerID="611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11" exitCode=0 Nov 03 17:36:37.906984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.906809 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" Nov 03 17:36:37.906984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.906849 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" event={"ID":"59800126-e27d-4638-879e-77c88cc5a506","Type":"ContainerDied","Data":"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11"} Nov 03 17:36:37.906984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.906899 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" event={"ID":"59800126-e27d-4638-879e-77c88cc5a506","Type":"ContainerDied","Data":"1c749f9e4960bc238782c12c084354e89f35303747893780c43ab98309955948"} Nov 03 17:36:37.906984 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.906918 6001 scope.go:122] "RemoveContainer" containerID="611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11" Nov 03 17:36:37.931121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.930984 6001 scope.go:122] "RemoveContainer" containerID="611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11" Nov 03 17:36:37.933587 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:37.933549 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11\": container with ID starting with 611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11 not found: ID does not exist" containerID="611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11" Nov 03 17:36:37.933821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.933788 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11"} err="failed to get container status \"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11\": rpc error: code = NotFound desc = could not find container \"611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11\": container with ID starting with 611160b2ae01458cd90ecb7880bb443fce2ea32534ef1b6c261ca32ab1ef3f11 not found: ID does not exist" Nov 03 17:36:37.947601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.945405 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:37.949732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:37.949685 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.050843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.050803 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:38.050843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.050842 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:38.051159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.050868 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\" (UniqueName: \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:38.051159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.050915 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:38.051159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.050938 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\" (UniqueName: \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\") pod \"59800126-e27d-4638-879e-77c88cc5a506\" (UID: \"59800126-e27d-4638-879e-77c88cc5a506\") " Nov 03 17:36:38.051159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.051145 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.051512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.051160 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.051512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.051413 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.051512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.051431 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.051512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.051478 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.053523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.053478 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.053689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.053672 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:38.053743 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.053708 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2" pod "59800126-e27d-4638-879e-77c88cc5a506" (UID: "59800126-e27d-4638-879e-77c88cc5a506"). InnerVolumeSpecName "kube-api-access-mtjs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:38.152976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.152888 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59800126-e27d-4638-879e-77c88cc5a506-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.152976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.152921 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59800126-e27d-4638-879e-77c88cc5a506-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.152976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.152931 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjs2\" (UniqueName: \"kubernetes.io/projected/59800126-e27d-4638-879e-77c88cc5a506-kube-api-access-mtjs2\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.184158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.184118 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx"] Nov 03 17:36:38.184277 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:36:38.184206 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5c1398_18fb_474c_8a83_ead621920c9c.slice/crio-d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a WatchSource:0}: Error finding container d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a: Status 404 returned error can't find the container with id d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a Nov 03 17:36:38.257149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.257115 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 17:36:38.269871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.269841 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt"] Nov 03 17:36:38.789353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.789320 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59800126-e27d-4638-879e-77c88cc5a506" path="/var/lib/kubelet/pods/59800126-e27d-4638-879e-77c88cc5a506/volumes" Nov 03 17:36:38.839960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.839914 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-w0-71d7b4-kserve-857c56657d-9h9zt" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:36:38.915814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.915772 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" event={"ID":"af5c1398-18fb-474c-8a83-ead621920c9c","Type":"ContainerStarted","Data":"9655790f31832a7dc2cd8c809691c0ee170cb59d275b36f3c7c5a8626011ec3a"} Nov 03 17:36:38.916053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.915843 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" event={"ID":"af5c1398-18fb-474c-8a83-ead621920c9c","Type":"ContainerStarted","Data":"d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a"} Nov 03 17:36:38.957860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:38.957797 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" podStartSLOduration=1.9577793 podStartE2EDuration="1.9577793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:38.957188088 +0000 UTC m=+4894.290401664" watchObservedRunningTime="2026-11-03 17:36:38.9577793 +0000 UTC m=+4894.290992870" Nov 03 17:36:39.925179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:39.925145 6001 generic.go:356] "Generic (PLEG): container finished" podID="af5c1398-18fb-474c-8a83-ead621920c9c" containerID="9655790f31832a7dc2cd8c809691c0ee170cb59d275b36f3c7c5a8626011ec3a" exitCode=0 Nov 03 17:36:39.925717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:39.925229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" event={"ID":"af5c1398-18fb-474c-8a83-ead621920c9c","Type":"ContainerDied","Data":"9655790f31832a7dc2cd8c809691c0ee170cb59d275b36f3c7c5a8626011ec3a"} Nov 03 17:36:40.623788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:40.623755 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:40.623999 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:40.623754 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:40.936970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:40.936941 6001 generic.go:356] "Generic (PLEG): container finished" podID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerID="d837c3bce145c6d7556f06d15bc8449db9a9599a343fec16a6d52c65bccba89b" exitCode=0 Nov 03 17:36:40.937342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:40.937024 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerDied","Data":"d837c3bce145c6d7556f06d15bc8449db9a9599a343fec16a6d52c65bccba89b"} Nov 03 17:36:40.937342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:40.937105 6001 scope.go:122] "RemoveContainer" containerID="3a24a22b4f97b88583ad6a11c3d4606d9d43d342609e3c73ac2e9ba019d68fe0" Nov 03 17:36:41.030391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.030359 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:36:41.083249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083220 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083290 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083411 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083489 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083531 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083554 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083582 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.083744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083602 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083663 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.083744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083717 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083751 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\") pod \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\" (UID: \"3250e537-243b-4b0b-bcb1-fa5bba0f02a8\") " Nov 03 17:36:41.083991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083819 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.083991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083960 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.084153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.083963 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:41.084254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.084233 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.084318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.084255 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.084318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.084264 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.084318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.084273 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.084318 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.084285 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.085856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.085824 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.085856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.085833 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:41.086097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.086081 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:41.086212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.086192 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r" pod "3250e537-243b-4b0b-bcb1-fa5bba0f02a8" (UID: "3250e537-243b-4b0b-bcb1-fa5bba0f02a8"). InnerVolumeSpecName "kube-api-access-9zk8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:41.126140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.126106 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:41.184986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.184903 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\") pod \"af5c1398-18fb-474c-8a83-ead621920c9c\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " Nov 03 17:36:41.185156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185048 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\" (UniqueName: \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\") pod \"af5c1398-18fb-474c-8a83-ead621920c9c\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " Nov 03 17:36:41.185156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185119 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\") pod \"af5c1398-18fb-474c-8a83-ead621920c9c\" (UID: \"af5c1398-18fb-474c-8a83-ead621920c9c\") " Nov 03 17:36:41.185567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185539 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca" pod "af5c1398-18fb-474c-8a83-ead621920c9c" (UID: "af5c1398-18fb-474c-8a83-ead621920c9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:41.185567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185554 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.185730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185594 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.185730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185612 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.185730 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.185628 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zk8r\" (UniqueName: \"kubernetes.io/projected/3250e537-243b-4b0b-bcb1-fa5bba0f02a8-kube-api-access-9zk8r\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.187068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.187049 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca" pod "af5c1398-18fb-474c-8a83-ead621920c9c" (UID: "af5c1398-18fb-474c-8a83-ead621920c9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:41.187144 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.187080 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r" pod "af5c1398-18fb-474c-8a83-ead621920c9c" (UID: "af5c1398-18fb-474c-8a83-ead621920c9c"). InnerVolumeSpecName "kube-api-access-74h5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:41.287164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.287128 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74h5r\" (UniqueName: \"kubernetes.io/projected/af5c1398-18fb-474c-8a83-ead621920c9c-kube-api-access-74h5r\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.287164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.287157 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.287164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.287168 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af5c1398-18fb-474c-8a83-ead621920c9c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.857117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.857087 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 17:36:41.887638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.887602 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:36:41.893958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.893935 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894023 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894113 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894157 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894178 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894200 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894222 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894279 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\") pod \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\" (UID: \"957bd4e6-a2de-4b74-94c7-c73e21c1534e\") " Nov 03 17:36:41.894425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894334 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.894578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894514 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.894663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894599 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.894724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894654 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:41.894824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894801 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.894894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894830 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.894894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894846 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.894894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894860 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.894894 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.894810 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.896413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.896380 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:41.896413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.896392 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "kube-api-access-zzqfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:41.896588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.896411 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:41.896588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.896554 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo" pod "957bd4e6-a2de-4b74-94c7-c73e21c1534e" (UID: "957bd4e6-a2de-4b74-94c7-c73e21c1534e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:41.945532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.945496 6001 generic.go:356] "Generic (PLEG): container finished" podID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerID="5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b" exitCode=0 Nov 03 17:36:41.946004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.945541 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" Nov 03 17:36:41.946004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.945573 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerDied","Data":"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b"} Nov 03 17:36:41.946004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.945626 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5" event={"ID":"957bd4e6-a2de-4b74-94c7-c73e21c1534e","Type":"ContainerDied","Data":"53729e57d0319ce6910cd3f4968630d404c7e74a0f1a2985af6dd5e5aa4917fb"} Nov 03 17:36:41.946004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.945649 6001 scope.go:122] "RemoveContainer" containerID="5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b" Nov 03 17:36:41.947599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.947570 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" Nov 03 17:36:41.947733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.947711 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s" event={"ID":"3250e537-243b-4b0b-bcb1-fa5bba0f02a8","Type":"ContainerDied","Data":"e364100db6ceb80971f0689183f0c3f90c851a416562bbc5cbb15ff4c1744e07"} Nov 03 17:36:41.949423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.949398 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" event={"ID":"af5c1398-18fb-474c-8a83-ead621920c9c","Type":"ContainerDied","Data":"d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a"} Nov 03 17:36:41.949617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.949459 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d443be0f26c0b6881aab5167e83198ada192c5db2d49d5b488ef51b9d0ed730a" Nov 03 17:36:41.949617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.949517 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx" Nov 03 17:36:41.970895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.970247 6001 scope.go:122] "RemoveContainer" containerID="897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661" Nov 03 17:36:41.982034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.982019 6001 scope.go:122] "RemoveContainer" containerID="5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b" Nov 03 17:36:41.982286 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:41.982269 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b\": container with ID starting with 5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b not found: ID does not exist" containerID="5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b" Nov 03 17:36:41.982376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.982294 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b"} err="failed to get container status \"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b\": rpc error: code = NotFound desc = could not find container \"5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b\": container with ID starting with 5ea2a89e62277b79a6e67a5b9cd776bd0eee9eed090cd3a754f4240fe4c8857b not found: ID does not exist" Nov 03 17:36:41.982376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.982310 6001 scope.go:122] "RemoveContainer" containerID="897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661" Nov 03 17:36:41.982602 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:41.982584 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661\": container with ID starting with 897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661 not found: ID does not exist" containerID="897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661" Nov 03 17:36:41.982661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.982616 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661"} err="failed to get container status \"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661\": rpc error: code = NotFound desc = could not find container \"897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661\": container with ID starting with 897747edbef39b60cb8473d6a42ba3f691f641751f7fac083be7208411873661 not found: ID does not exist" Nov 03 17:36:41.982661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.982633 6001 scope.go:122] "RemoveContainer" containerID="d837c3bce145c6d7556f06d15bc8449db9a9599a343fec16a6d52c65bccba89b" Nov 03 17:36:41.996265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.996236 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.996265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.996263 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.996425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.996277 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqfn\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-kube-api-access-zzqfn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.996425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.996286 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/957bd4e6-a2de-4b74-94c7-c73e21c1534e-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:41.996425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:41.996295 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/957bd4e6-a2de-4b74-94c7-c73e21c1534e-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:42.004723 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.004685 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 17:36:42.012119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.012089 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-c6f303-openshift-default-756b94f7b7-gkgk5"] Nov 03 17:36:42.065511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.065429 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:36:42.067815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.067781 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-z7d5s"] Nov 03 17:36:42.093132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.093037 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 17:36:42.789728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.789697 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" path="/var/lib/kubelet/pods/3250e537-243b-4b0b-bcb1-fa5bba0f02a8/volumes" Nov 03 17:36:42.790230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:42.790216 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" path="/var/lib/kubelet/pods/957bd4e6-a2de-4b74-94c7-c73e21c1534e/volumes" Nov 03 17:36:44.865917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:44.865874 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:45.859740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.859707 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 17:36:45.947040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.946987 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\") pod \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " Nov 03 17:36:45.947596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.947070 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\" (UniqueName: \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\") pod \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\" (UID: \"bfafb4ea-c764-48d7-9f9f-f7a458f662ed\") " Nov 03 17:36:45.947596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.947526 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume" pod "bfafb4ea-c764-48d7-9f9f-f7a458f662ed" (UID: "bfafb4ea-c764-48d7-9f9f-f7a458f662ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:45.947724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.947666 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.950087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.950057 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g" pod "bfafb4ea-c764-48d7-9f9f-f7a458f662ed" (UID: "bfafb4ea-c764-48d7-9f9f-f7a458f662ed"). InnerVolumeSpecName "kube-api-access-bh77g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:45.982107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.982063 6001 generic.go:356] "Generic (PLEG): container finished" podID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerID="bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783" exitCode=0 Nov 03 17:36:45.982107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.982115 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" Nov 03 17:36:45.982352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.982151 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" event={"ID":"bfafb4ea-c764-48d7-9f9f-f7a458f662ed","Type":"ContainerDied","Data":"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783"} Nov 03 17:36:45.982352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.982212 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm" event={"ID":"bfafb4ea-c764-48d7-9f9f-f7a458f662ed","Type":"ContainerDied","Data":"06e8409555b282c93f99932ea8790d4001d2514af02d8e4bcdab63865471bb51"} Nov 03 17:36:45.982352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.982235 6001 scope.go:122] "RemoveContainer" containerID="bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783" Nov 03 17:36:45.994768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.994737 6001 scope.go:122] "RemoveContainer" containerID="bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783" Nov 03 17:36:45.995111 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:45.995091 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783\": container with ID starting with bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783 not found: ID does not exist" containerID="bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783" Nov 03 17:36:45.995166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:45.995121 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783"} err="failed to get container status \"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783\": rpc error: code = NotFound desc = could not find container \"bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783\": container with ID starting with bc16a1fb902dfb32cd7c59ce837a468bcda5182b14eda42134088f4c937ac783 not found: ID does not exist" Nov 03 17:36:46.042957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:46.042915 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 17:36:46.048548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:46.048516 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh77g\" (UniqueName: \"kubernetes.io/projected/bfafb4ea-c764-48d7-9f9f-f7a458f662ed-kube-api-access-bh77g\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:46.049813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:46.049789 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c6f303-fd4c8d787-xgtbm"] Nov 03 17:36:46.789636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:46.789600 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" path="/var/lib/kubelet/pods/bfafb4ea-c764-48d7-9f9f-f7a458f662ed/volumes" Nov 03 17:36:46.888848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:46.888804 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" containerID="cri-o://0a7ea6f8f3a450936cf47270434599148c19b614464d2ab9e3ee923158bb8239" gracePeriod=25 Nov 03 17:36:48.899436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899400 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb"] Nov 03 17:36:48.899887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899872 6001 cpu_manager.go:414] "containerMap: removing container" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:36:48.899938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899888 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:36:48.899938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899903 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" Nov 03 17:36:48.899938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899909 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" Nov 03 17:36:48.899938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899928 6001 cpu_manager.go:414] "containerMap: removing container" podUID="af5c1398-18fb-474c-8a83-ead621920c9c" containerName="revoke-keys" Nov 03 17:36:48.899938 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899933 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af5c1398-18fb-474c-8a83-ead621920c9c" containerName="revoke-keys" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899945 6001 cpu_manager.go:414] "containerMap: removing container" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899950 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899972 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899977 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899984 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.899989 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900048 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="af5c1398-18fb-474c-8a83-ead621920c9c" containerName="revoke-keys" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900056 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="59800126-e27d-4638-879e-77c88cc5a506" containerName="main" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900068 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900074 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3250e537-243b-4b0b-bcb1-fa5bba0f02a8" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900080 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900087 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:36:48.900102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.900093 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfafb4ea-c764-48d7-9f9f-f7a458f662ed" containerName="payload-pre-processing" Nov 03 17:36:48.903183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.903166 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:48.921882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.921857 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:36:48.921882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.921869 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:36:48.934540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.934520 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:36:48.934689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.934554 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:36:48.938400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.938378 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb"] Nov 03 17:36:48.980967 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.980938 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwjqf\" (UniqueName: \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:48.981131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.981110 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:48.981203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:48.981150 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.082201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.082166 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.082201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.082206 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.082522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.082267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwjqf\" (UniqueName: \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.082972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.082945 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.084781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.084762 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.106688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.106655 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwjqf\" (UniqueName: \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\") pod \"maas-api-revoke-sub-2bc362d1-dclbb\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.216487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.216353 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:49.497702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.497644 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 17:36:49.520094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.520060 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb"] Nov 03 17:36:49.520230 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:36:49.520161 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161a8cb9_a40d_4ea0_a776_6a07ecc3cb06.slice/crio-2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6 WatchSource:0}: Error finding container 2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6: Status 404 returned error can't find the container with id 2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6 Nov 03 17:36:49.532659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:49.532624 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 17:36:50.012009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:50.011909 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" event={"ID":"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06","Type":"ContainerStarted","Data":"6af24d009381ddec2e87967755756e5e4a196f7a3fdf3517c5e7826883237415"} Nov 03 17:36:50.012009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:50.011968 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" event={"ID":"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06","Type":"ContainerStarted","Data":"2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6"} Nov 03 17:36:50.058589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:50.058311 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" podStartSLOduration=2.058291015 podStartE2EDuration="2.058291015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:36:50.053187537 +0000 UTC m=+4905.386401107" watchObservedRunningTime="2026-11-03 17:36:50.058291015 +0000 UTC m=+4905.391504584" Nov 03 17:36:50.623686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:50.623649 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:50.623893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:50.623681 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:51.020175 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:51.020139 6001 generic.go:356] "Generic (PLEG): container finished" podID="161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" containerID="6af24d009381ddec2e87967755756e5e4a196f7a3fdf3517c5e7826883237415" exitCode=0 Nov 03 17:36:51.020649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:51.020202 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" event={"ID":"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06","Type":"ContainerDied","Data":"6af24d009381ddec2e87967755756e5e4a196f7a3fdf3517c5e7826883237415"} Nov 03 17:36:51.300293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:51.300192 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:36:51.300530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:51.300482 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" containerID="cri-o://2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c" gracePeriod=30 Nov 03 17:36:51.468869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:51.468824 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:52.170590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.170554 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:52.250664 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:52.250627 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:36:52.323519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.323397 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\" (UniqueName: \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\") pod \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " Nov 03 17:36:52.323692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.323587 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\") pod \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " Nov 03 17:36:52.323692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.323660 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\") pod \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\" (UID: \"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06\") " Nov 03 17:36:52.324070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.324044 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca" pod "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" (UID: "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:52.325663 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.325636 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca" pod "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" (UID: "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:52.325780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.325706 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf" pod "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" (UID: "161a8cb9-a40d-4ea0-a776-6a07ecc3cb06"). InnerVolumeSpecName "kube-api-access-lwjqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:52.424860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.424823 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:52.424860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.424853 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:52.424860 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:52.424864 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwjqf\" (UniqueName: \"kubernetes.io/projected/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06-kube-api-access-lwjqf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:53.033900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.033852 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" event={"ID":"161a8cb9-a40d-4ea0-a776-6a07ecc3cb06","Type":"ContainerDied","Data":"2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6"} Nov 03 17:36:53.034086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.033926 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e361d51d5f0d407c13242ab4a0d7acbf50fdc5c522f5d4204a4bee6beb529a6" Nov 03 17:36:53.034086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.033862 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb" Nov 03 17:36:53.821361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.821335 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 17:36:53.868088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.868055 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 17:36:53.950178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950148 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\" (UniqueName: \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\") pod \"6f153cb0-a64a-460f-8510-f3680f0d9b87\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " Nov 03 17:36:53.950356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950283 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\" (UniqueName: \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\") pod \"1745b37d-cab1-4761-81ff-83656d6490a6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " Nov 03 17:36:53.950356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950306 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\") pod \"1745b37d-cab1-4761-81ff-83656d6490a6\" (UID: \"1745b37d-cab1-4761-81ff-83656d6490a6\") " Nov 03 17:36:53.950356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950342 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\") pod \"6f153cb0-a64a-460f-8510-f3680f0d9b87\" (UID: \"6f153cb0-a64a-460f-8510-f3680f0d9b87\") " Nov 03 17:36:53.950772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950742 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume" pod "1745b37d-cab1-4761-81ff-83656d6490a6" (UID: "1745b37d-cab1-4761-81ff-83656d6490a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:53.950898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950800 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume" pod "6f153cb0-a64a-460f-8510-f3680f0d9b87" (UID: "6f153cb0-a64a-460f-8510-f3680f0d9b87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:53.950980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950947 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1745b37d-cab1-4761-81ff-83656d6490a6-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:53.950980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.950968 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f153cb0-a64a-460f-8510-f3680f0d9b87-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:53.952278 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.952254 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz" pod "6f153cb0-a64a-460f-8510-f3680f0d9b87" (UID: "6f153cb0-a64a-460f-8510-f3680f0d9b87"). InnerVolumeSpecName "kube-api-access-nt6nz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:53.952388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:53.952372 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78" pod "1745b37d-cab1-4761-81ff-83656d6490a6" (UID: "1745b37d-cab1-4761-81ff-83656d6490a6"). InnerVolumeSpecName "kube-api-access-nvt78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:54.041853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.041758 6001 generic.go:356] "Generic (PLEG): container finished" podID="1745b37d-cab1-4761-81ff-83656d6490a6" containerID="eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e" exitCode=0 Nov 03 17:36:54.041853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.041813 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" Nov 03 17:36:54.042098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.041843 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" event={"ID":"1745b37d-cab1-4761-81ff-83656d6490a6","Type":"ContainerDied","Data":"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e"} Nov 03 17:36:54.042098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.041901 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6" event={"ID":"1745b37d-cab1-4761-81ff-83656d6490a6","Type":"ContainerDied","Data":"d768eeabcdb3c745343f34b99b7ff87eb616bbc70aa0efba7b4a521b0f42a15b"} Nov 03 17:36:54.042098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.041923 6001 scope.go:122] "RemoveContainer" containerID="eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e" Nov 03 17:36:54.043419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.043398 6001 generic.go:356] "Generic (PLEG): container finished" podID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerID="3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f" exitCode=0 Nov 03 17:36:54.043568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.043479 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" Nov 03 17:36:54.043568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.043476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" event={"ID":"6f153cb0-a64a-460f-8510-f3680f0d9b87","Type":"ContainerDied","Data":"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f"} Nov 03 17:36:54.043693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.043621 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl" event={"ID":"6f153cb0-a64a-460f-8510-f3680f0d9b87","Type":"ContainerDied","Data":"b4d7f7b910a9911bd01b328c9c0d9a898bd0a1ea0b63d78294f27b22196c6ff6"} Nov 03 17:36:54.051880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.051852 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6nz\" (UniqueName: \"kubernetes.io/projected/6f153cb0-a64a-460f-8510-f3680f0d9b87-kube-api-access-nt6nz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:54.051995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.051881 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvt78\" (UniqueName: \"kubernetes.io/projected/1745b37d-cab1-4761-81ff-83656d6490a6-kube-api-access-nvt78\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:54.053776 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.053761 6001 scope.go:122] "RemoveContainer" containerID="eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e" Nov 03 17:36:54.054046 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:54.054029 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e\": container with ID starting with eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e not found: ID does not exist" containerID="eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e" Nov 03 17:36:54.054119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.054052 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e"} err="failed to get container status \"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e\": rpc error: code = NotFound desc = could not find container \"eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e\": container with ID starting with eebe690080be272aba46be1fcb12bdf58d72be80c8ca72cfc18781c4c6a6675e not found: ID does not exist" Nov 03 17:36:54.054119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.054067 6001 scope.go:122] "RemoveContainer" containerID="3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f" Nov 03 17:36:54.064069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.064051 6001 scope.go:122] "RemoveContainer" containerID="3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f" Nov 03 17:36:54.064376 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:54.064349 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f\": container with ID starting with 3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f not found: ID does not exist" containerID="3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f" Nov 03 17:36:54.064432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.064386 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f"} err="failed to get container status \"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f\": rpc error: code = NotFound desc = could not find container \"3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f\": container with ID starting with 3acf479b1603809264139ca302a5f5d3b2c3af61189a1af9a5e827c2ccd1408f not found: ID does not exist" Nov 03 17:36:54.097524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.097500 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 17:36:54.100191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.100170 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-71d7b4-7767d64467-2q8vl"] Nov 03 17:36:54.136537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.136510 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 17:36:54.139352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.139333 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-71d7b4-589dcf787c-w42l6"] Nov 03 17:36:54.498336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.498287 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" containerID="cri-o://53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be" gracePeriod=25 Nov 03 17:36:54.533951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.533878 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" containerID="cri-o://80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a" gracePeriod=25 Nov 03 17:36:54.789796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.789712 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" path="/var/lib/kubelet/pods/1745b37d-cab1-4761-81ff-83656d6490a6/volumes" Nov 03 17:36:54.790131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:54.790118 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" path="/var/lib/kubelet/pods/6f153cb0-a64a-460f-8510-f3680f0d9b87/volumes" Nov 03 17:36:55.589686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:55.589646 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:55.590215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:55.589707 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:56.576950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.576880 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:36:56.684375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684235 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684347 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684414 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684486 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684525 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684592 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684652 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684685 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.684954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.684721 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\") pod \"7383014e-2260-45d8-ae51-cfc249d6a476\" (UID: \"7383014e-2260-45d8-ae51-cfc249d6a476\") " Nov 03 17:36:56.685588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.685426 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:56.685741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.685719 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.686282 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.686669 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.686740 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.689635 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.690102 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.691864 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "kube-api-access-rdtjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:56.698466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.695274 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy" pod "7383014e-2260-45d8-ae51-cfc249d6a476" (UID: "7383014e-2260-45d8-ae51-cfc249d6a476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:56.786637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786596 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786637 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786637 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786652 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdtjb\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-kube-api-access-rdtjb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786666 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786680 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7383014e-2260-45d8-ae51-cfc249d6a476-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786693 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7383014e-2260-45d8-ae51-cfc249d6a476-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786704 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7383014e-2260-45d8-ae51-cfc249d6a476-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786719 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:56.786867 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:56.786733 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7383014e-2260-45d8-ae51-cfc249d6a476-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.068074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.068026 6001 generic.go:356] "Generic (PLEG): container finished" podID="7383014e-2260-45d8-ae51-cfc249d6a476" containerID="2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c" exitCode=0 Nov 03 17:36:57.068303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.068079 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" Nov 03 17:36:57.068303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.068109 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerDied","Data":"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c"} Nov 03 17:36:57.068303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.068185 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn" event={"ID":"7383014e-2260-45d8-ae51-cfc249d6a476","Type":"ContainerDied","Data":"5fe49523634c20630379fe95540814180ad6980786b5345b00964165f8a912ca"} Nov 03 17:36:57.068303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.068216 6001 scope.go:122] "RemoveContainer" containerID="2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c" Nov 03 17:36:57.080946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.080926 6001 scope.go:122] "RemoveContainer" containerID="1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b" Nov 03 17:36:57.091478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.091434 6001 scope.go:122] "RemoveContainer" containerID="2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c" Nov 03 17:36:57.091784 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:57.091765 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c\": container with ID starting with 2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c not found: ID does not exist" containerID="2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c" Nov 03 17:36:57.091865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.091792 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c"} err="failed to get container status \"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c\": rpc error: code = NotFound desc = could not find container \"2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c\": container with ID starting with 2b3f6dc2693cda8fc2a9864561613aa4eb909a07cd76bef0078730c3c3014a0c not found: ID does not exist" Nov 03 17:36:57.091865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.091810 6001 scope.go:122] "RemoveContainer" containerID="1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b" Nov 03 17:36:57.092055 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:57.092038 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b\": container with ID starting with 1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b not found: ID does not exist" containerID="1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b" Nov 03 17:36:57.092114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.092061 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b"} err="failed to get container status \"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b\": rpc error: code = NotFound desc = could not find container \"1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b\": container with ID starting with 1d8738db3834ffd11d3c34f11c71facb651f168fe613da97209b6717deccce8b not found: ID does not exist" Nov 03 17:36:57.114533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.114496 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" containerID="cri-o://d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e" gracePeriod=45 Nov 03 17:36:57.126066 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.126040 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:36:57.138836 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.138809 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-790111-openshift-default-976c8fb97-kc5fn"] Nov 03 17:36:57.412124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.412090 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:36:57.595890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.595809 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.595958 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\" (UniqueName: \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596043 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.595982 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596045 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\" (UniqueName: \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596082 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596115 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\") pod \"52a28929-28b8-42a3-b06a-a2a5851a8847\" (UID: \"52a28929-28b8-42a3-b06a-a2a5851a8847\") " Nov 03 17:36:57.596348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596321 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.596562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596541 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.596654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596545 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.596654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.596591 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.598837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.598810 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.598955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.598815 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "kube-api-access-ftqgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:57.598955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.598876 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs" pod "52a28929-28b8-42a3-b06a-a2a5851a8847" (UID: "52a28929-28b8-42a3-b06a-a2a5851a8847"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:57.697859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.697818 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.697859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.697855 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52a28929-28b8-42a3-b06a-a2a5851a8847-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.698468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.697870 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftqgk\" (UniqueName: \"kubernetes.io/projected/52a28929-28b8-42a3-b06a-a2a5851a8847-kube-api-access-ftqgk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.698468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.697908 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.698468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:57.697923 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52a28929-28b8-42a3-b06a-a2a5851a8847-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:36:58.078142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.078110 6001 generic.go:356] "Generic (PLEG): container finished" podID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerID="d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e" exitCode=0 Nov 03 17:36:58.078352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.078170 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" Nov 03 17:36:58.078352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.078191 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" event={"ID":"52a28929-28b8-42a3-b06a-a2a5851a8847","Type":"ContainerDied","Data":"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e"} Nov 03 17:36:58.078352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.078235 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6" event={"ID":"52a28929-28b8-42a3-b06a-a2a5851a8847","Type":"ContainerDied","Data":"c41bf3fa05bb5069406d00467f3e8312d68553407d7ee6bd1e78039780178d46"} Nov 03 17:36:58.078352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.078254 6001 scope.go:122] "RemoveContainer" containerID="d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e" Nov 03 17:36:58.099463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.098273 6001 scope.go:122] "RemoveContainer" containerID="d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e" Nov 03 17:36:58.107464 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:36:58.103550 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e\": container with ID starting with d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e not found: ID does not exist" containerID="d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e" Nov 03 17:36:58.107464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.103591 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e"} err="failed to get container status \"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e\": rpc error: code = NotFound desc = could not find container \"d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e\": container with ID starting with d1e2d5b5ca0d47208865ec5405a74fd2d429270138353768f3f248278834996e not found: ID does not exist" Nov 03 17:36:58.169371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.168497 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 17:36:58.194469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.190702 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-460104/premium-w4-460104-kserve-5f5775c9c-cxfc6"] Nov 03 17:36:58.789094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.789061 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" path="/var/lib/kubelet/pods/52a28929-28b8-42a3-b06a-a2a5851a8847/volumes" Nov 03 17:36:58.789561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:36:58.789545 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" path="/var/lib/kubelet/pods/7383014e-2260-45d8-ae51-cfc249d6a476/volumes" Nov 03 17:37:01.471574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:01.471534 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:05.590188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:05.590152 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:05.590599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:05.590154 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:06.030090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:06.030055 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8"] Nov 03 17:37:06.042428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:06.042392 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf208a7-d72p8"] Nov 03 17:37:06.788298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:06.788263 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068415a3-cd0f-4b74-9c9c-6a1684da815d" path="/var/lib/kubelet/pods/068415a3-cd0f-4b74-9c9c-6a1684da815d/volumes" Nov 03 17:37:07.141238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.141210 6001 generic.go:356] "Generic (PLEG): container finished" podID="d5567754-676a-4298-9dd7-50ccfab12a68" containerID="0a7ea6f8f3a450936cf47270434599148c19b614464d2ab9e3ee923158bb8239" exitCode=0 Nov 03 17:37:07.141367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.141290 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" event={"ID":"d5567754-676a-4298-9dd7-50ccfab12a68","Type":"ContainerDied","Data":"0a7ea6f8f3a450936cf47270434599148c19b614464d2ab9e3ee923158bb8239"} Nov 03 17:37:07.141367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.141349 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" event={"ID":"d5567754-676a-4298-9dd7-50ccfab12a68","Type":"ContainerDied","Data":"eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981"} Nov 03 17:37:07.141485 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.141367 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb55f1cf090042000b20fcee3aba12eb956c86fc3667e8561f4f045f4d60f981" Nov 03 17:37:07.155205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.155180 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:37:07.210280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.210240 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\") pod \"d5567754-676a-4298-9dd7-50ccfab12a68\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " Nov 03 17:37:07.210500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.210355 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\" (UniqueName: \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\") pod \"d5567754-676a-4298-9dd7-50ccfab12a68\" (UID: \"d5567754-676a-4298-9dd7-50ccfab12a68\") " Nov 03 17:37:07.210771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.210732 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume" pod "d5567754-676a-4298-9dd7-50ccfab12a68" (UID: "d5567754-676a-4298-9dd7-50ccfab12a68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:07.213067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.213023 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f" pod "d5567754-676a-4298-9dd7-50ccfab12a68" (UID: "d5567754-676a-4298-9dd7-50ccfab12a68"). InnerVolumeSpecName "kube-api-access-wmj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:07.312252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.312159 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5567754-676a-4298-9dd7-50ccfab12a68-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:07.312252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:07.312191 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmj9f\" (UniqueName: \"kubernetes.io/projected/d5567754-676a-4298-9dd7-50ccfab12a68-kube-api-access-wmj9f\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:08.148081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.148045 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc" Nov 03 17:37:08.201258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.201213 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:37:08.213887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.213851 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-790111-75f47dd547-mjzsc"] Nov 03 17:37:08.306393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.306356 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 17:37:08.306647 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.306594 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" containerID="cri-o://015c35bb4129ad99cf832b380b72f461b98cd7546f8c246d24a9e5fda24a771d" gracePeriod=30 Nov 03 17:37:08.788085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:08.788046 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" path="/var/lib/kubelet/pods/d5567754-676a-4298-9dd7-50ccfab12a68/volumes" Nov 03 17:37:13.577944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.577876 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:37:13.681794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681755 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.681976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681816 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.681976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681864 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.681976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681913 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.681976 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681962 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.682177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.681991 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.682177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682033 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.682177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682111 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.682177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682122 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.682369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682238 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\") pod \"3844c53f-0103-4055-8a03-9416753e16e4\" (UID: \"3844c53f-0103-4055-8a03-9416753e16e4\") " Nov 03 17:37:13.682369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682272 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.682640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682585 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.682768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682651 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:13.682844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682796 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.682844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682821 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.682844 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682837 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.683044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682851 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3844c53f-0103-4055-8a03-9416753e16e4-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.683044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.682857 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.684397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.684368 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:13.684605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.684559 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "kube-api-access-mcr4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:13.684665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.684614 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.684665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.684648 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token" pod "3844c53f-0103-4055-8a03-9416753e16e4" (UID: "3844c53f-0103-4055-8a03-9416753e16e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:13.784941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.784902 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3844c53f-0103-4055-8a03-9416753e16e4-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.784941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.784936 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcr4t\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-kube-api-access-mcr4t\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.784941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.784951 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3844c53f-0103-4055-8a03-9416753e16e4-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.785207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.784963 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.785207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.784975 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3844c53f-0103-4055-8a03-9416753e16e4-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.959810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.959728 6001 scope.go:122] "RemoveContainer" containerID="a23f33a7c105dfd468e2cae5c29114cf78bed136cc04039d0680fda5d48201f3" Nov 03 17:37:13.969700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.969676 6001 scope.go:122] "RemoveContainer" containerID="0e354216adbad57e9e1302256588b77aeb4ebb787a327b6e384e3706d472ae56" Nov 03 17:37:13.980106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:13.980083 6001 scope.go:122] "RemoveContainer" containerID="015c35bb4129ad99cf832b380b72f461b98cd7546f8c246d24a9e5fda24a771d" Nov 03 17:37:14.015419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.015402 6001 scope.go:122] "RemoveContainer" containerID="0a7ea6f8f3a450936cf47270434599148c19b614464d2ab9e3ee923158bb8239" Nov 03 17:37:14.193476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.193405 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerDied","Data":"015c35bb4129ad99cf832b380b72f461b98cd7546f8c246d24a9e5fda24a771d"} Nov 03 17:37:14.193687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.193506 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" event={"ID":"3844c53f-0103-4055-8a03-9416753e16e4","Type":"ContainerDied","Data":"6a8e5feeb23e914978a6cf686b33434f73404d8cd6dfb21cb3745606971ec916"} Nov 03 17:37:14.193687 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.193521 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w" Nov 03 17:37:14.293607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.293564 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 17:37:14.299051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.299021 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-snj9w"] Nov 03 17:37:14.789954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.789915 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3844c53f-0103-4055-8a03-9416753e16e4" path="/var/lib/kubelet/pods/3844c53f-0103-4055-8a03-9416753e16e4/volumes" Nov 03 17:37:14.797162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.797133 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 17:37:14.825307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.825232 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 17:37:14.898630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.898589 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\" (UniqueName: \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\") pod \"95372900-df8f-4882-b20d-f589727dc7b4\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " Nov 03 17:37:14.898847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.898641 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\" (UniqueName: \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\") pod \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " Nov 03 17:37:14.898847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.898673 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\") pod \"95372900-df8f-4882-b20d-f589727dc7b4\" (UID: \"95372900-df8f-4882-b20d-f589727dc7b4\") " Nov 03 17:37:14.898847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.898755 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\") pod \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\" (UID: \"3bbe941a-2ad2-4c94-9d3a-dad871031c3b\") " Nov 03 17:37:14.899489 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.899129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume" pod "95372900-df8f-4882-b20d-f589727dc7b4" (UID: "95372900-df8f-4882-b20d-f589727dc7b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:14.899677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.899581 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume" pod "3bbe941a-2ad2-4c94-9d3a-dad871031c3b" (UID: "3bbe941a-2ad2-4c94-9d3a-dad871031c3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:14.901002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.900976 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48" pod "95372900-df8f-4882-b20d-f589727dc7b4" (UID: "95372900-df8f-4882-b20d-f589727dc7b4"). InnerVolumeSpecName "kube-api-access-dvs48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:14.901123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:14.901080 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj" pod "3bbe941a-2ad2-4c94-9d3a-dad871031c3b" (UID: "3bbe941a-2ad2-4c94-9d3a-dad871031c3b"). InnerVolumeSpecName "kube-api-access-9shnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:15.000143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.000111 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.000143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.000139 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvs48\" (UniqueName: \"kubernetes.io/projected/95372900-df8f-4882-b20d-f589727dc7b4-kube-api-access-dvs48\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.000143 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.000150 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9shnj\" (UniqueName: \"kubernetes.io/projected/3bbe941a-2ad2-4c94-9d3a-dad871031c3b-kube-api-access-9shnj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.000406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.000160 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95372900-df8f-4882-b20d-f589727dc7b4-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.200795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.200698 6001 generic.go:356] "Generic (PLEG): container finished" podID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerID="53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be" exitCode=0 Nov 03 17:37:15.200795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.200738 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" event={"ID":"3bbe941a-2ad2-4c94-9d3a-dad871031c3b","Type":"ContainerDied","Data":"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be"} Nov 03 17:37:15.200795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.200771 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" Nov 03 17:37:15.201107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.200812 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd" event={"ID":"3bbe941a-2ad2-4c94-9d3a-dad871031c3b","Type":"ContainerDied","Data":"35f05874923752053875279d0c7a9c474590a125b5f7b897b5555b2362b40539"} Nov 03 17:37:15.201107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.200845 6001 scope.go:122] "RemoveContainer" containerID="53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be" Nov 03 17:37:15.202575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.202549 6001 generic.go:356] "Generic (PLEG): container finished" podID="95372900-df8f-4882-b20d-f589727dc7b4" containerID="80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a" exitCode=0 Nov 03 17:37:15.202698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.202581 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" event={"ID":"95372900-df8f-4882-b20d-f589727dc7b4","Type":"ContainerDied","Data":"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a"} Nov 03 17:37:15.202698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.202617 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" Nov 03 17:37:15.202698 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.202660 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p" event={"ID":"95372900-df8f-4882-b20d-f589727dc7b4","Type":"ContainerDied","Data":"e5e65d0959b3bdb35604df7add15466bc3f58e591bf66fd344b1e0d41da0c979"} Nov 03 17:37:15.211839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.211740 6001 scope.go:122] "RemoveContainer" containerID="53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be" Nov 03 17:37:15.212228 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:37:15.212206 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be\": container with ID starting with 53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be not found: ID does not exist" containerID="53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be" Nov 03 17:37:15.212299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.212239 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be"} err="failed to get container status \"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be\": rpc error: code = NotFound desc = could not find container \"53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be\": container with ID starting with 53d78f22b92df59266fe0c612a99767dcbf874a575148db25654065e375107be not found: ID does not exist" Nov 03 17:37:15.212299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.212261 6001 scope.go:122] "RemoveContainer" containerID="80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a" Nov 03 17:37:15.222382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.222357 6001 scope.go:122] "RemoveContainer" containerID="80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a" Nov 03 17:37:15.222737 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:37:15.222714 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a\": container with ID starting with 80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a not found: ID does not exist" containerID="80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a" Nov 03 17:37:15.222821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.222747 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a"} err="failed to get container status \"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a\": rpc error: code = NotFound desc = could not find container \"80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a\": container with ID starting with 80c46f34bfab1e7f7f6f76e9ea7027cec8aed54c15680b4dfa497a561601bb1a not found: ID does not exist" Nov 03 17:37:15.264315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.264280 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 17:37:15.275311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.275278 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 17:37:15.275551 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.275513 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" podUID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" containerName="discovery" containerID="cri-o://d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39" gracePeriod=30 Nov 03 17:37:15.277715 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.277689 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-460104-5ff68b8c4c-72m9p"] Nov 03 17:37:15.320593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.320555 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 17:37:15.322805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.322780 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-460104-846dc69785-jdvpd"] Nov 03 17:37:15.537640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.537614 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 17:37:15.606505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606475 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606530 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606645 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606725 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606700 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606758 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606862 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.606929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.606916 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\") pod \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\" (UID: \"8bb05426-b20c-4e5c-b4f5-0c4b44430808\") " Nov 03 17:37:15.607150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.607127 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:15.607321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.607302 6001 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-ca-configmap\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.609522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609364 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:15.609522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609435 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:15.609522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609461 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "kube-api-access-jkz7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:15.609780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609757 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:15.609861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609843 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:15.609914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.609864 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts" pod "8bb05426-b20c-4e5c-b4f5-0c4b44430808" (UID: "8bb05426-b20c-4e5c-b4f5-0c4b44430808"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:15.708932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708892 6001 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-cacerts\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.708932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708927 6001 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-kubeconfig\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.708932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708938 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz7n\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-kube-api-access-jkz7n\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.709192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708947 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.709192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708956 6001 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8bb05426-b20c-4e5c-b4f5-0c4b44430808-istio-csr-dns-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:15.709192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:15.708967 6001 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb05426-b20c-4e5c-b4f5-0c4b44430808-local-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:16.209916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.209883 6001 generic.go:356] "Generic (PLEG): container finished" podID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" containerID="d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39" exitCode=0 Nov 03 17:37:16.210426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.209936 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" Nov 03 17:37:16.210426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.209943 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" event={"ID":"8bb05426-b20c-4e5c-b4f5-0c4b44430808","Type":"ContainerDied","Data":"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39"} Nov 03 17:37:16.210426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.209997 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2" event={"ID":"8bb05426-b20c-4e5c-b4f5-0c4b44430808","Type":"ContainerDied","Data":"da4707745f0da8efdffefb9d2cf406d41c0eaca9a96c675347988bd792d0625b"} Nov 03 17:37:16.210426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.210018 6001 scope.go:122] "RemoveContainer" containerID="d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39" Nov 03 17:37:16.226145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.226121 6001 scope.go:122] "RemoveContainer" containerID="d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39" Nov 03 17:37:16.226523 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:37:16.226495 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39\": container with ID starting with d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39 not found: ID does not exist" containerID="d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39" Nov 03 17:37:16.226658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.226532 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39"} err="failed to get container status \"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39\": rpc error: code = NotFound desc = could not find container \"d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39\": container with ID starting with d227dba6fab1e452ef32f32281a7a105cd75e8435dac8baeb57df32e15f68e39 not found: ID does not exist" Nov 03 17:37:16.266631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.266513 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 17:37:16.279169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.279136 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-p7wz2"] Nov 03 17:37:16.789148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.789107 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" path="/var/lib/kubelet/pods/3bbe941a-2ad2-4c94-9d3a-dad871031c3b/volumes" Nov 03 17:37:16.789614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.789600 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" path="/var/lib/kubelet/pods/8bb05426-b20c-4e5c-b4f5-0c4b44430808/volumes" Nov 03 17:37:16.789989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:16.789976 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95372900-df8f-4882-b20d-f589727dc7b4" path="/var/lib/kubelet/pods/95372900-df8f-4882-b20d-f589727dc7b4/volumes" Nov 03 17:37:22.251647 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:37:22.251608 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:37:25.621679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.621641 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf"] Nov 03 17:37:25.622112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622087 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" Nov 03 17:37:25.622112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622099 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" Nov 03 17:37:25.622112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622111 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622117 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622128 6001 cpu_manager.go:414] "containerMap: removing container" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622134 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957bd4e6-a2de-4b74-94c7-c73e21c1534e" containerName="istio-proxy" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622153 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622159 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622167 6001 cpu_manager.go:414] "containerMap: removing container" podUID="161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" containerName="revoke-keys" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622171 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" containerName="revoke-keys" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622177 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622184 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622194 6001 cpu_manager.go:414] "containerMap: removing container" podUID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" containerName="discovery" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622199 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" containerName="discovery" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622210 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622215 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622225 6001 cpu_manager.go:414] "containerMap: removing container" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" Nov 03 17:37:25.622224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622231 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622240 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622244 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622252 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622257 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622276 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622281 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622294 6001 cpu_manager.go:414] "containerMap: removing container" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622301 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622368 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622378 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" containerName="revoke-keys" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622387 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f153cb0-a64a-460f-8510-f3680f0d9b87" containerName="payload-pre-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622393 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb05426-b20c-4e5c-b4f5-0c4b44430808" containerName="discovery" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622399 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5567754-676a-4298-9dd7-50ccfab12a68" containerName="payload-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622406 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a28929-28b8-42a3-b06a-a2a5851a8847" containerName="main" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622413 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622419 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7383014e-2260-45d8-ae51-cfc249d6a476" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622427 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3844c53f-0103-4055-8a03-9416753e16e4" containerName="istio-proxy" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622433 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="95372900-df8f-4882-b20d-f589727dc7b4" containerName="payload-pre-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622439 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1745b37d-cab1-4761-81ff-83656d6490a6" containerName="payload-processing" Nov 03 17:37:25.622734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.622462 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bbe941a-2ad2-4c94-9d3a-dad871031c3b" containerName="payload-processing" Nov 03 17:37:25.625411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.625394 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.643215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.643191 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:37:25.643537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.643509 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:37:25.655961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.655937 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:37:25.656225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.656207 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:37:25.660075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.660048 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf"] Nov 03 17:37:25.713376 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.713335 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.713608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.713404 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b86sz\" (UniqueName: \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.713608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.713524 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.814712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.814665 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.814944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.814735 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b86sz\" (UniqueName: \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.814944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.814772 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.815500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.815426 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.817270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.817249 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.838500 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.838463 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b86sz\" (UniqueName: \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\") pod \"maas-api-revoke-sub-dcb3e408-vf2xf\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:25.938131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:25.938044 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:26.030385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:26.030352 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt"] Nov 03 17:37:26.032470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:26.032434 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-803a5dd4-48vgt"] Nov 03 17:37:26.245835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:26.245804 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf"] Nov 03 17:37:26.246657 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:37:26.246629 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b6b0804_b261_449d_a4e3_d40e34f1ce49.slice/crio-a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8 WatchSource:0}: Error finding container a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8: Status 404 returned error can't find the container with id a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8 Nov 03 17:37:26.276722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:26.276691 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" event={"ID":"1b6b0804-b261-449d-a4e3-d40e34f1ce49","Type":"ContainerStarted","Data":"a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8"} Nov 03 17:37:26.788591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:26.788553 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70e316f-065c-457d-8623-9e4e892f7eab" path="/var/lib/kubelet/pods/c70e316f-065c-457d-8623-9e4e892f7eab/volumes" Nov 03 17:37:27.284048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:27.284015 6001 generic.go:356] "Generic (PLEG): container finished" podID="1b6b0804-b261-449d-a4e3-d40e34f1ce49" containerID="e74b5b9e776e166505268bfb74d849f0b84736e8d8274c1e484e71695c6665f6" exitCode=0 Nov 03 17:37:27.284234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:27.284102 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" event={"ID":"1b6b0804-b261-449d-a4e3-d40e34f1ce49","Type":"ContainerDied","Data":"e74b5b9e776e166505268bfb74d849f0b84736e8d8274c1e484e71695c6665f6"} Nov 03 17:37:28.429995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.429961 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:28.543147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.543056 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\") pod \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " Nov 03 17:37:28.543147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.543097 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\") pod \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " Nov 03 17:37:28.543373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.543220 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\" (UniqueName: \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\") pod \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\" (UID: \"1b6b0804-b261-449d-a4e3-d40e34f1ce49\") " Nov 03 17:37:28.543596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.543566 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca" pod "1b6b0804-b261-449d-a4e3-d40e34f1ce49" (UID: "1b6b0804-b261-449d-a4e3-d40e34f1ce49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:28.545367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.545337 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca" pod "1b6b0804-b261-449d-a4e3-d40e34f1ce49" (UID: "1b6b0804-b261-449d-a4e3-d40e34f1ce49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:28.545548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.545377 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz" pod "1b6b0804-b261-449d-a4e3-d40e34f1ce49" (UID: "1b6b0804-b261-449d-a4e3-d40e34f1ce49"). InnerVolumeSpecName "kube-api-access-b86sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:28.644113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.644080 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.644113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.644111 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b6b0804-b261-449d-a4e3-d40e34f1ce49-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.644113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:28.644121 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b86sz\" (UniqueName: \"kubernetes.io/projected/1b6b0804-b261-449d-a4e3-d40e34f1ce49-kube-api-access-b86sz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:29.298902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:29.298869 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" Nov 03 17:37:29.298902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:29.298893 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf" event={"ID":"1b6b0804-b261-449d-a4e3-d40e34f1ce49","Type":"ContainerDied","Data":"a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8"} Nov 03 17:37:29.299147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:29.298974 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a470bb5814568d9e631c4694f6fbb5f1ad2f15613aed144729fb6d9d003cbfe8" Nov 03 17:37:30.210952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:30.210915 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 17:37:42.953429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.953353 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d"] Nov 03 17:37:42.953877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.953845 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1b6b0804-b261-449d-a4e3-d40e34f1ce49" containerName="revoke-keys" Nov 03 17:37:42.953877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.953862 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b6b0804-b261-449d-a4e3-d40e34f1ce49" containerName="revoke-keys" Nov 03 17:37:42.953958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.953950 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b6b0804-b261-449d-a4e3-d40e34f1ce49" containerName="revoke-keys" Nov 03 17:37:42.957191 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.957173 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:42.980299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.980261 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl8fb\" (UniqueName: \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:42.980491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.980412 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:42.980605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.980581 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:42.984409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.984382 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:37:42.992885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.992858 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:37:42.993054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.992949 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:37:42.993054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.992960 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:37:42.995648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:42.995621 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d"] Nov 03 17:37:43.081672 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.081632 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.081903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.081687 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl8fb\" (UniqueName: \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.081903 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.081752 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.082425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.082401 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.084356 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.084331 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.111135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.111098 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl8fb\" (UniqueName: \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\") pod \"maas-api-revoke-sub-b3609fe8-dqx4d\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.271506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.271465 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:43.578224 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:43.578189 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d"] Nov 03 17:37:43.578394 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:37:43.578369 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded59f61_f739_4a97_be3f_8cd30370db6c.slice/crio-c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6 WatchSource:0}: Error finding container c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6: Status 404 returned error can't find the container with id c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6 Nov 03 17:37:44.385626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:44.385592 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" event={"ID":"ded59f61-f739-4a97-be3f-8cd30370db6c","Type":"ContainerStarted","Data":"8a9070598e27f181d784dc99ceda2bd7ba11d889bbea883dde5eba6a7a09762e"} Nov 03 17:37:44.386036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:44.385646 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" event={"ID":"ded59f61-f739-4a97-be3f-8cd30370db6c","Type":"ContainerStarted","Data":"c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6"} Nov 03 17:37:44.430432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:44.430377 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" podStartSLOduration=2.43036171 podStartE2EDuration="2.43036171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:44.42970736 +0000 UTC m=+4959.762920929" watchObservedRunningTime="2026-11-03 17:37:44.43036171 +0000 UTC m=+4959.763575278" Nov 03 17:37:45.232002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.231957 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" containerID="cri-o://4e18f543f0d987cae7499a0f55448416f25c2bd098aa2fe7010237104ba0d248" gracePeriod=45 Nov 03 17:37:45.393368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.393331 6001 generic.go:356] "Generic (PLEG): container finished" podID="7d725e50-1018-495a-b24d-42e60e376e61" containerID="4e18f543f0d987cae7499a0f55448416f25c2bd098aa2fe7010237104ba0d248" exitCode=0 Nov 03 17:37:45.393897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.393408 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" event={"ID":"7d725e50-1018-495a-b24d-42e60e376e61","Type":"ContainerDied","Data":"4e18f543f0d987cae7499a0f55448416f25c2bd098aa2fe7010237104ba0d248"} Nov 03 17:37:45.395275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.395246 6001 generic.go:356] "Generic (PLEG): container finished" podID="ded59f61-f739-4a97-be3f-8cd30370db6c" containerID="8a9070598e27f181d784dc99ceda2bd7ba11d889bbea883dde5eba6a7a09762e" exitCode=0 Nov 03 17:37:45.395417 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.395281 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" event={"ID":"ded59f61-f739-4a97-be3f-8cd30370db6c","Type":"ContainerDied","Data":"8a9070598e27f181d784dc99ceda2bd7ba11d889bbea883dde5eba6a7a09762e"} Nov 03 17:37:45.498912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.498839 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:37:45.608243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608208 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608268 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608297 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\" (UniqueName: \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608339 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608389 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608481 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\") pod \"7d725e50-1018-495a-b24d-42e60e376e61\" (UID: \"7d725e50-1018-495a-b24d-42e60e376e61\") " Nov 03 17:37:45.608726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608617 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.608854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608730 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.608910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608871 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.609027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608937 6001 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-tmp-dir\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.609027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.608952 6001 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-home\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.610704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.610676 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.610975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.610950 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:45.611089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.611015 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm" pod "7d725e50-1018-495a-b24d-42e60e376e61" (UID: "7d725e50-1018-495a-b24d-42e60e376e61"). InnerVolumeSpecName "kube-api-access-9rggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:45.710283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.710233 6001 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-dshm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.710283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.710276 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rggm\" (UniqueName: \"kubernetes.io/projected/7d725e50-1018-495a-b24d-42e60e376e61-kube-api-access-9rggm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.710283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.710291 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d725e50-1018-495a-b24d-42e60e376e61-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.710683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:45.710305 6001 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d725e50-1018-495a-b24d-42e60e376e61-model-cache\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:46.403108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.403020 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" Nov 03 17:37:46.403108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.403059 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc" event={"ID":"7d725e50-1018-495a-b24d-42e60e376e61","Type":"ContainerDied","Data":"10b3906c2b918606877140585a8d1be551c1cf3a3c22c18c6a1a665069f260e2"} Nov 03 17:37:46.403670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.403113 6001 scope.go:122] "RemoveContainer" containerID="4e18f543f0d987cae7499a0f55448416f25c2bd098aa2fe7010237104ba0d248" Nov 03 17:37:46.459317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.459274 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 17:37:46.471689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.471655 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-491905/simulator-w1-491905-kserve-7bc7dc8b86-5j5bc"] Nov 03 17:37:46.557559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.557530 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:46.618728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.618677 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\" (UniqueName: \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\") pod \"ded59f61-f739-4a97-be3f-8cd30370db6c\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " Nov 03 17:37:46.618960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.618833 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\") pod \"ded59f61-f739-4a97-be3f-8cd30370db6c\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " Nov 03 17:37:46.618960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.618911 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\") pod \"ded59f61-f739-4a97-be3f-8cd30370db6c\" (UID: \"ded59f61-f739-4a97-be3f-8cd30370db6c\") " Nov 03 17:37:46.619228 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.619197 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca" pod "ded59f61-f739-4a97-be3f-8cd30370db6c" (UID: "ded59f61-f739-4a97-be3f-8cd30370db6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:46.619393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.619365 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:46.621393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.621348 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca" pod "ded59f61-f739-4a97-be3f-8cd30370db6c" (UID: "ded59f61-f739-4a97-be3f-8cd30370db6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:46.621653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.621363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb" pod "ded59f61-f739-4a97-be3f-8cd30370db6c" (UID: "ded59f61-f739-4a97-be3f-8cd30370db6c"). InnerVolumeSpecName "kube-api-access-nl8fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:46.720771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.720680 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl8fb\" (UniqueName: \"kubernetes.io/projected/ded59f61-f739-4a97-be3f-8cd30370db6c-kube-api-access-nl8fb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:46.720771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.720719 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ded59f61-f739-4a97-be3f-8cd30370db6c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:37:46.790138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:46.790104 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d725e50-1018-495a-b24d-42e60e376e61" path="/var/lib/kubelet/pods/7d725e50-1018-495a-b24d-42e60e376e61/volumes" Nov 03 17:37:47.412239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:47.412209 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" Nov 03 17:37:47.412239 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:47.412229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d" event={"ID":"ded59f61-f739-4a97-be3f-8cd30370db6c","Type":"ContainerDied","Data":"c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6"} Nov 03 17:37:47.412855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:47.412291 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5c0efa95b4ef8316758de18b1521ccd00a7cff1d2bd7471c1ac5e1d322261c6" Nov 03 17:37:52.250641 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:37:52.250595 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:37:54.038987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:54.038951 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm"] Nov 03 17:37:54.050898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:54.050872 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d191c67-pgpcm"] Nov 03 17:37:54.788314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:54.788281 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5d4136-a787-464e-8621-b9aebc35e22c" path="/var/lib/kubelet/pods/ef5d4136-a787-464e-8621-b9aebc35e22c/volumes" Nov 03 17:37:55.224845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.224816 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225334 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225354 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225392 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ded59f61-f739-4a97-be3f-8cd30370db6c" containerName="revoke-keys" Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225399 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded59f61-f739-4a97-be3f-8cd30370db6c" containerName="revoke-keys" Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225511 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded59f61-f739-4a97-be3f-8cd30370db6c" containerName="revoke-keys" Nov 03 17:37:55.225829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.225529 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d725e50-1018-495a-b24d-42e60e376e61" containerName="main" Nov 03 17:37:55.228817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.228798 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.247370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.247352 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-491905-openshift-default-dockercfg-dc6wb\"" Nov 03 17:37:55.263783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.263756 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:37:55.304972 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.304944 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.304977 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnxk\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305086 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305104 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305130 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305198 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305229 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305298 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.305415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.305344 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406253 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406300 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnxk\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406352 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406370 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406394 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406412 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406427 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406487 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406784 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406845 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.406997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.406944 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.407103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.407044 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.407244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.407225 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.408945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.408914 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.409122 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.409103 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.430053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.430028 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.430242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.430222 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnxk\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.542004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.541911 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:55.863245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.863215 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:37:55.863578 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:37:55.863554 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e6bf4d_ee7c_41e5_aa18_da5ae0f3c70c.slice/crio-ba93c9cc6b66557a4dcd037da21a7d533c5e9b123e038f51f47a05289d688912 WatchSource:0}: Error finding container ba93c9cc6b66557a4dcd037da21a7d533c5e9b123e038f51f47a05289d688912: Status 404 returned error can't find the container with id ba93c9cc6b66557a4dcd037da21a7d533c5e9b123e038f51f47a05289d688912 Nov 03 17:37:55.865943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.865910 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:37:55.866025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.865981 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:37:55.866025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:55.866011 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:37:56.472609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.472567 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" event={"ID":"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c","Type":"ContainerStarted","Data":"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053"} Nov 03 17:37:56.473108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.472640 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" event={"ID":"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c","Type":"ContainerStarted","Data":"ba93c9cc6b66557a4dcd037da21a7d533c5e9b123e038f51f47a05289d688912"} Nov 03 17:37:56.542957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.542922 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:37:56.566685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.566642 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:37:56.567004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.566975 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:37:56.569464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:56.567421 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podStartSLOduration=1.567404343 podStartE2EDuration="1.567404343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:37:56.531200478 +0000 UTC m=+4971.864414047" watchObservedRunningTime="2026-11-03 17:37:56.567404343 +0000 UTC m=+4971.900617914" Nov 03 17:37:57.543246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:57.543204 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:37:57.543810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:57.543264 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:37:58.543193 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:58.543154 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:37:58.543423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:58.543207 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:37:59.542369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:59.542331 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:37:59.542598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:37:59.542391 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:38:00.543003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:00.542966 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:38:00.543587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:00.543032 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:38:01.542575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:01.542539 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:38:01.542809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:01.542599 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:38:02.543128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:02.543089 6001 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" start-of-body= Nov 03 17:38:02.543599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:02.543157 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.172:15021/healthz/ready\": dial tcp 10.134.1.172:15021: connect: connection refused" Nov 03 17:38:04.036064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.036028 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z"] Nov 03 17:38:04.048210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.048166 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3024a083-9fd5z"] Nov 03 17:38:04.496825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.496792 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:38:04.497098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.497079 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:38:04.498053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.498034 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:38:04.801548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:04.801423 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0" path="/var/lib/kubelet/pods/7891b4dd-bf08-48f8-ab54-45b3ebf7d6a0/volumes" Nov 03 17:38:10.116203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:10.116167 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:38:10.116880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:10.116366 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" containerID="cri-o://4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053" gracePeriod=30 Nov 03 17:38:11.141400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:11.141360 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4c8d89d8-khtcc_7ac3d41f-22e1-4ef3-8389-4fc3e73d5196/authorino/0.log" Nov 03 17:38:11.793042 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:11.793007 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zd2c8_cb8bbc2c-4a30-45a4-a981-63996002b0c4/kuadrant-console-plugin/0.log" Nov 03 17:38:14.097070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:14.097036 6001 scope.go:122] "RemoveContainer" containerID="85f1b92a788718fedbe1942e9a07a8cf80fbc207169174dc9698772b38344ea0" Nov 03 17:38:14.107141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:14.107121 6001 scope.go:122] "RemoveContainer" containerID="5159bd2c33115818785d90d59b78127385990a3567962ea3f216d811b1937981" Nov 03 17:38:15.031955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.031917 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf"] Nov 03 17:38:15.043689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.043658 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1c0412-pgnkf"] Nov 03 17:38:15.383634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.383607 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:38:15.430732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430673 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430740 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430790 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430829 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430887 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430958 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.430998 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.431051 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.431346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.431105 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\") pod \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\" (UID: \"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c\") " Nov 03 17:38:15.432249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.432223 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.432702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.432367 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.432702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.432601 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.432904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.432882 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:15.433944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.433917 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.434577 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.434533 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.434702 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.434683 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:15.435878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.435857 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "kube-api-access-zmnxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:15.436095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.436073 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo" pod "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" (UID: "25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:38:15.532584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532556 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532582 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532591 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532602 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532610 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmnxk\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-kube-api-access-zmnxk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532620 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532632 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532639 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.532780 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.532648 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.596509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.596384 6001 generic.go:356] "Generic (PLEG): container finished" podID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerID="4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053" exitCode=0 Nov 03 17:38:15.596509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.596434 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" Nov 03 17:38:15.596509 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.596465 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" event={"ID":"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c","Type":"ContainerDied","Data":"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053"} Nov 03 17:38:15.596814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.596520 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd" event={"ID":"25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c","Type":"ContainerDied","Data":"ba93c9cc6b66557a4dcd037da21a7d533c5e9b123e038f51f47a05289d688912"} Nov 03 17:38:15.596814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.596552 6001 scope.go:122] "RemoveContainer" containerID="4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053" Nov 03 17:38:15.606660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.606639 6001 scope.go:122] "RemoveContainer" containerID="4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053" Nov 03 17:38:15.606933 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:15.606909 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053\": container with ID starting with 4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053 not found: ID does not exist" containerID="4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053" Nov 03 17:38:15.606977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.606943 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053"} err="failed to get container status \"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053\": rpc error: code = NotFound desc = could not find container \"4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053\": container with ID starting with 4ea3aa67d4271747f37ed2b58605692a29f73b4ab308d4c33bd4adafe4f86053 not found: ID does not exist" Nov 03 17:38:15.649717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.649601 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:38:15.654335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:15.654303 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-kgndd"] Nov 03 17:38:16.788030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:16.787997 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" path="/var/lib/kubelet/pods/25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c/volumes" Nov 03 17:38:16.788458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:16.788432 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a97d99-587b-492a-a958-384284488439" path="/var/lib/kubelet/pods/b6a97d99-587b-492a-a958-384284488439/volumes" Nov 03 17:38:22.250169 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:22.250131 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:38:25.039764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.039724 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg"] Nov 03 17:38:25.049640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.049609 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-493b4e6d-vn5rg"] Nov 03 17:38:25.278222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.278183 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:25.278770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.278753 6001 cpu_manager.go:414] "containerMap: removing container" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" Nov 03 17:38:25.278820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.278772 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" Nov 03 17:38:25.278868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.278849 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="25e6bf4d-ee7c-41e5-aa18-da5ae0f3c70c" containerName="istio-proxy" Nov 03 17:38:25.282044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.282026 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.297413 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.297349 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-184babf9\"" Nov 03 17:38:25.297582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.297558 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-184babf9-dockercfg-s4p82\"" Nov 03 17:38:25.308660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.308628 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:25.326741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.326693 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.326981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.326827 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzmw\" (UniqueName: \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.390493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.390437 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:25.394815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.394790 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.416667 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.416633 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:25.428153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.428119 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzmw\" (UniqueName: \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.428336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.428191 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.428924 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.428902 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.452546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.452513 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzmw\" (UniqueName: \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\") pod \"payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.528973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.528936 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4sv\" (UniqueName: \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.528973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.528985 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.595675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.595588 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:25.630118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.630079 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4sv\" (UniqueName: \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.630595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.630130 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.630871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.630849 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.657881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.657843 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4sv\" (UniqueName: \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\") pod \"payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.708050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.708016 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:25.899503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:25.899439 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:25.899812 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:38:25.899787 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f840c2_331a_47e6_a979_aeb28c81c446.slice/crio-462a24b8b4f959d8a61d8f6341d58b99664966db5ee1306214051020ee014f89 WatchSource:0}: Error finding container 462a24b8b4f959d8a61d8f6341d58b99664966db5ee1306214051020ee014f89: Status 404 returned error can't find the container with id 462a24b8b4f959d8a61d8f6341d58b99664966db5ee1306214051020ee014f89 Nov 03 17:38:26.014138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.014109 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:26.014705 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:38:26.014675 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37fbba8_d994_49a1_a2b7_f2b657f42266.slice/crio-b75542b7cf932fb522885181434880976b2348a8fa6c5a39d754e2ec4327d301 WatchSource:0}: Error finding container b75542b7cf932fb522885181434880976b2348a8fa6c5a39d754e2ec4327d301: Status 404 returned error can't find the container with id b75542b7cf932fb522885181434880976b2348a8fa6c5a39d754e2ec4327d301 Nov 03 17:38:26.663208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.663170 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" event={"ID":"a37fbba8-d994-49a1-a2b7-f2b657f42266","Type":"ContainerStarted","Data":"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898"} Nov 03 17:38:26.663712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.663239 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" event={"ID":"a37fbba8-d994-49a1-a2b7-f2b657f42266","Type":"ContainerStarted","Data":"b75542b7cf932fb522885181434880976b2348a8fa6c5a39d754e2ec4327d301"} Nov 03 17:38:26.663712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.663347 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:26.664834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.664814 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" event={"ID":"15f840c2-331a-47e6-a979-aeb28c81c446","Type":"ContainerStarted","Data":"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16"} Nov 03 17:38:26.664956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.664847 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" event={"ID":"15f840c2-331a-47e6-a979-aeb28c81c446","Type":"ContainerStarted","Data":"462a24b8b4f959d8a61d8f6341d58b99664966db5ee1306214051020ee014f89"} Nov 03 17:38:26.664956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.664867 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:26.706329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.706224 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" podStartSLOduration=1.706206947 podStartE2EDuration="1.706206947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:26.705151422 +0000 UTC m=+5002.038364990" watchObservedRunningTime="2026-11-03 17:38:26.706206947 +0000 UTC m=+5002.039420515" Nov 03 17:38:26.747965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.747914 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" podStartSLOduration=1.747900247 podStartE2EDuration="1.747900247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:26.746867276 +0000 UTC m=+5002.080080859" watchObservedRunningTime="2026-11-03 17:38:26.747900247 +0000 UTC m=+5002.081113815" Nov 03 17:38:26.787801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:26.787762 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="868453dd-ed95-4ce3-b8e4-d809021c2c40" path="/var/lib/kubelet/pods/868453dd-ed95-4ce3-b8e4-d809021c2c40/volumes" Nov 03 17:38:37.673468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:37.673418 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:37.674113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:37.673526 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:39.309918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.309867 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8"] Nov 03 17:38:39.314896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.314869 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.335595 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.335559 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8"] Nov 03 17:38:39.344717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.344672 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:38:39.344926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.344692 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:38:39.344926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.344694 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:38:39.344926 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.344783 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:38:39.472996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.472951 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.473238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.473086 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.473238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.473157 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.574875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.574769 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.574875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.574860 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.575089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.574956 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.575696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.575677 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.577342 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:39.577312 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:39.604095 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:39.604062 6001 projected.go:196] Error preparing data for projected volume kube-api-access-5rv24 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 17:38:39.604292 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:39.604132 6001 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24 podName:9c78ef8f-7859-4fc2-ab3a-5b42d182660d nodeName:}" failed. No retries permitted until 2026-11-03 17:38:40.104116968 +0000 UTC m=+5015.437330515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5rv24" (UniqueName: "kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24") pod "maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" (UID: "9c78ef8f-7859-4fc2-ab3a-5b42d182660d") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Nov 03 17:38:40.181003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.180962 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:40.205295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.205254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") pod \"maas-api-revoke-keys-e2e-ait-184babf9-rtlb8\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:40.229478 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.229427 6001 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" secret="" err="secret \"maas-api-cleanup-dockercfg-7q7j9\" not found" Nov 03 17:38:40.229683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.229517 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:40.530602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.530574 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8"] Nov 03 17:38:40.532173 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:38:40.532141 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c78ef8f_7859_4fc2_ab3a_5b42d182660d.slice/crio-04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb WatchSource:0}: Error finding container 04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb: Status 404 returned error can't find the container with id 04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb Nov 03 17:38:40.753301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:40.753264 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" event={"ID":"9c78ef8f-7859-4fc2-ab3a-5b42d182660d","Type":"ContainerStarted","Data":"04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb"} Nov 03 17:38:41.760996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:41.760962 6001 generic.go:356] "Generic (PLEG): container finished" podID="9c78ef8f-7859-4fc2-ab3a-5b42d182660d" containerID="99cc9caf893a51907bfa3ac8ca28c4621880ebf84689008f2f3986dbe8ea97a9" exitCode=0 Nov 03 17:38:41.761459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:41.761054 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" event={"ID":"9c78ef8f-7859-4fc2-ab3a-5b42d182660d","Type":"ContainerDied","Data":"99cc9caf893a51907bfa3ac8ca28c4621880ebf84689008f2f3986dbe8ea97a9"} Nov 03 17:38:42.903958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:42.903928 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:43.012650 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.012607 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") pod \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " Nov 03 17:38:43.012875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.012679 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\") pod \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " Nov 03 17:38:43.012875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.012703 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\") pod \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\" (UID: \"9c78ef8f-7859-4fc2-ab3a-5b42d182660d\") " Nov 03 17:38:43.013219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.013054 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca" pod "9c78ef8f-7859-4fc2-ab3a-5b42d182660d" (UID: "9c78ef8f-7859-4fc2-ab3a-5b42d182660d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:43.014940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.014911 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24" pod "9c78ef8f-7859-4fc2-ab3a-5b42d182660d" (UID: "9c78ef8f-7859-4fc2-ab3a-5b42d182660d"). InnerVolumeSpecName "kube-api-access-5rv24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:43.015063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.014943 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca" pod "9c78ef8f-7859-4fc2-ab3a-5b42d182660d" (UID: "9c78ef8f-7859-4fc2-ab3a-5b42d182660d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:43.114458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.114411 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rv24\" (UniqueName: \"kubernetes.io/projected/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-kube-api-access-5rv24\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:43.114639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.114485 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:43.114639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.114496 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c78ef8f-7859-4fc2-ab3a-5b42d182660d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:43.774490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.774431 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" Nov 03 17:38:43.774490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.774479 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8" event={"ID":"9c78ef8f-7859-4fc2-ab3a-5b42d182660d","Type":"ContainerDied","Data":"04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb"} Nov 03 17:38:43.774762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:43.774524 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04139e470f2515b4b8e0e8dc34dde12e35f79e973d1eaf18782fd85a1573d7bb" Nov 03 17:38:49.730961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:49.730924 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:49.759960 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:49.759925 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:51.935806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:51.935764 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8"] Nov 03 17:38:51.948495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:51.948457 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-184babf9-rtlb8"] Nov 03 17:38:52.250493 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:52.250435 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:38:52.788296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:52.788261 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c78ef8f-7859-4fc2-ab3a-5b42d182660d" path="/var/lib/kubelet/pods/9c78ef8f-7859-4fc2-ab3a-5b42d182660d/volumes" Nov 03 17:38:54.731519 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:54.731477 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" podUID="a37fbba8-d994-49a1-a2b7-f2b657f42266" containerName="payload-processing" containerID="cri-o://3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898" gracePeriod=25 Nov 03 17:38:54.761023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:54.760946 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" podUID="15f840c2-331a-47e6-a979-aeb28c81c446" containerName="payload-pre-processing" containerID="cri-o://8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16" gracePeriod=25 Nov 03 17:38:55.017279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.017253 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:55.034153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.034126 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\") pod \"a37fbba8-d994-49a1-a2b7-f2b657f42266\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " Nov 03 17:38:55.034279 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.034218 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\" (UniqueName: \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\") pod \"a37fbba8-d994-49a1-a2b7-f2b657f42266\" (UID: \"a37fbba8-d994-49a1-a2b7-f2b657f42266\") " Nov 03 17:38:55.034556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.034529 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume" pod "a37fbba8-d994-49a1-a2b7-f2b657f42266" (UID: "a37fbba8-d994-49a1-a2b7-f2b657f42266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:55.036544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.036519 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv" pod "a37fbba8-d994-49a1-a2b7-f2b657f42266" (UID: "a37fbba8-d994-49a1-a2b7-f2b657f42266"). InnerVolumeSpecName "kube-api-access-4f4sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:55.045011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.044990 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:55.135027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.134983 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\" (UniqueName: \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\") pod \"15f840c2-331a-47e6-a979-aeb28c81c446\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " Nov 03 17:38:55.135196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.135096 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\") pod \"15f840c2-331a-47e6-a979-aeb28c81c446\" (UID: \"15f840c2-331a-47e6-a979-aeb28c81c446\") " Nov 03 17:38:55.135506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.135476 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume" pod "15f840c2-331a-47e6-a979-aeb28c81c446" (UID: "15f840c2-331a-47e6-a979-aeb28c81c446"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:55.135506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.135490 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a37fbba8-d994-49a1-a2b7-f2b657f42266-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:55.135686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.135527 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4sv\" (UniqueName: \"kubernetes.io/projected/a37fbba8-d994-49a1-a2b7-f2b657f42266-kube-api-access-4f4sv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:55.137114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.137092 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw" pod "15f840c2-331a-47e6-a979-aeb28c81c446" (UID: "15f840c2-331a-47e6-a979-aeb28c81c446"). InnerVolumeSpecName "kube-api-access-gvzmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:55.237047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.237005 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvzmw\" (UniqueName: \"kubernetes.io/projected/15f840c2-331a-47e6-a979-aeb28c81c446-kube-api-access-gvzmw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:55.237047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.237041 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f840c2-331a-47e6-a979-aeb28c81c446-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:38:55.843153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.843116 6001 generic.go:356] "Generic (PLEG): container finished" podID="a37fbba8-d994-49a1-a2b7-f2b657f42266" containerID="3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898" exitCode=0 Nov 03 17:38:55.843740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.843164 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" Nov 03 17:38:55.843740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.843203 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" event={"ID":"a37fbba8-d994-49a1-a2b7-f2b657f42266","Type":"ContainerDied","Data":"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898"} Nov 03 17:38:55.843740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.843269 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz" event={"ID":"a37fbba8-d994-49a1-a2b7-f2b657f42266","Type":"ContainerDied","Data":"b75542b7cf932fb522885181434880976b2348a8fa6c5a39d754e2ec4327d301"} Nov 03 17:38:55.843740 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.843298 6001 scope.go:122] "RemoveContainer" containerID="3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898" Nov 03 17:38:55.845142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.845119 6001 generic.go:356] "Generic (PLEG): container finished" podID="15f840c2-331a-47e6-a979-aeb28c81c446" containerID="8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16" exitCode=0 Nov 03 17:38:55.845274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.845164 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" Nov 03 17:38:55.845274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.845205 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" event={"ID":"15f840c2-331a-47e6-a979-aeb28c81c446","Type":"ContainerDied","Data":"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16"} Nov 03 17:38:55.845274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.845258 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td" event={"ID":"15f840c2-331a-47e6-a979-aeb28c81c446","Type":"ContainerDied","Data":"462a24b8b4f959d8a61d8f6341d58b99664966db5ee1306214051020ee014f89"} Nov 03 17:38:55.854392 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.854374 6001 scope.go:122] "RemoveContainer" containerID="3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898" Nov 03 17:38:55.854777 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:55.854719 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898\": container with ID starting with 3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898 not found: ID does not exist" containerID="3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898" Nov 03 17:38:55.854777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.854755 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898"} err="failed to get container status \"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898\": rpc error: code = NotFound desc = could not find container \"3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898\": container with ID starting with 3fa17763ff2eff6d38afd7a15aa5472fecf1b271fa6e9320303bf228cafe0898 not found: ID does not exist" Nov 03 17:38:55.854897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.854782 6001 scope.go:122] "RemoveContainer" containerID="8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16" Nov 03 17:38:55.863613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.863593 6001 scope.go:122] "RemoveContainer" containerID="8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16" Nov 03 17:38:55.863900 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:38:55.863878 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16\": container with ID starting with 8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16 not found: ID does not exist" containerID="8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16" Nov 03 17:38:55.863953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.863907 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16"} err="failed to get container status \"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16\": rpc error: code = NotFound desc = could not find container \"8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16\": container with ID starting with 8d8ba3d82984ef732d2b9b74ca325def17d4167e05da9579328b071ec1474e16 not found: ID does not exist" Nov 03 17:38:55.909826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.909790 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:55.922032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.921999 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-184babf9-6c54b7cbd5-scxkz"] Nov 03 17:38:55.959713 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.959676 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:55.964750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:55.964721 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-184babf9-659bf749f9-w67td"] Nov 03 17:38:56.788384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:56.788350 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f840c2-331a-47e6-a979-aeb28c81c446" path="/var/lib/kubelet/pods/15f840c2-331a-47e6-a979-aeb28c81c446/volumes" Nov 03 17:38:56.788814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:38:56.788799 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a37fbba8-d994-49a1-a2b7-f2b657f42266" path="/var/lib/kubelet/pods/a37fbba8-d994-49a1-a2b7-f2b657f42266/volumes" Nov 03 17:39:14.159460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:14.159410 6001 scope.go:122] "RemoveContainer" containerID="7ff539b16bd419b5544354a203d92d88af9bdd83d0834aeda959f9a8f9df85e3" Nov 03 17:39:14.169295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:14.169270 6001 scope.go:122] "RemoveContainer" containerID="929a4095194709a848d3b278a801a76b5371f714832af92b481f55b9f06f78c3" Nov 03 17:39:14.178814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:14.178791 6001 scope.go:122] "RemoveContainer" containerID="70f1e16e57ec2d1434b933304fc65c7a8253ad6f885c5931c969631f1f330a5e" Nov 03 17:39:14.187986 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:14.187960 6001 scope.go:122] "RemoveContainer" containerID="111dcc2004f25e3aa56d43f4602e94900c652634eaa2dfdb2cc69adb82014096" Nov 03 17:39:20.643365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.643274 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8"] Nov 03 17:39:20.643958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.643936 6001 cpu_manager.go:414] "containerMap: removing container" podUID="15f840c2-331a-47e6-a979-aeb28c81c446" containerName="payload-pre-processing" Nov 03 17:39:20.644049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.643960 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f840c2-331a-47e6-a979-aeb28c81c446" containerName="payload-pre-processing" Nov 03 17:39:20.644049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.643979 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9c78ef8f-7859-4fc2-ab3a-5b42d182660d" containerName="revoke-keys" Nov 03 17:39:20.644049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.643988 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c78ef8f-7859-4fc2-ab3a-5b42d182660d" containerName="revoke-keys" Nov 03 17:39:20.644049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.644000 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a37fbba8-d994-49a1-a2b7-f2b657f42266" containerName="payload-processing" Nov 03 17:39:20.644049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.644007 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a37fbba8-d994-49a1-a2b7-f2b657f42266" containerName="payload-processing" Nov 03 17:39:20.644327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.644138 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f840c2-331a-47e6-a979-aeb28c81c446" containerName="payload-pre-processing" Nov 03 17:39:20.644327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.644150 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c78ef8f-7859-4fc2-ab3a-5b42d182660d" containerName="revoke-keys" Nov 03 17:39:20.644327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.644160 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a37fbba8-d994-49a1-a2b7-f2b657f42266" containerName="payload-processing" Nov 03 17:39:20.647406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.647379 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.672709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.672681 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8"] Nov 03 17:39:20.705744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.705703 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n4j\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-kube-api-access-s9n4j\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.705961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.705755 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.705961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.705881 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.705961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.705933 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.705961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.705954 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.706129 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.706041 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.706129 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.706089 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.706208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.706139 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.706244 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.706219 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n4j\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-kube-api-access-s9n4j\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807188 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807240 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807281 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807420 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807494 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807556 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807592 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807648 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807725 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807788 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.807991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807879 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.808082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.807986 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.808319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.808297 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.809778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.809756 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.810183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.810161 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.832306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.832269 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.832423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.832355 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n4j\" (UniqueName: \"kubernetes.io/projected/90e58ef7-2fe2-4060-af83-3d06ba7cbea9-kube-api-access-s9n4j\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-bgct8\" (UID: \"90e58ef7-2fe2-4060-af83-3d06ba7cbea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:20.961126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:20.961030 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:21.284374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.284347 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8"] Nov 03 17:39:21.284570 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:39:21.284544 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e58ef7_2fe2_4060_af83_3d06ba7cbea9.slice/crio-bba2bccafdbfacffcc66d19805d0f6f7bc2a8735b7c9511f95bf36cd8c8203ad WatchSource:0}: Error finding container bba2bccafdbfacffcc66d19805d0f6f7bc2a8735b7c9511f95bf36cd8c8203ad: Status 404 returned error can't find the container with id bba2bccafdbfacffcc66d19805d0f6f7bc2a8735b7c9511f95bf36cd8c8203ad Nov 03 17:39:21.286823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.286792 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:21.286890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.286855 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:21.286890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.286885 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:21.996477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.996424 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" event={"ID":"90e58ef7-2fe2-4060-af83-3d06ba7cbea9","Type":"ContainerStarted","Data":"d40ac3054bd7e9e18a11f7c32370827e66eeb390b3eeeaae351965d349b15f72"} Nov 03 17:39:21.996911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:21.996505 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" event={"ID":"90e58ef7-2fe2-4060-af83-3d06ba7cbea9","Type":"ContainerStarted","Data":"bba2bccafdbfacffcc66d19805d0f6f7bc2a8735b7c9511f95bf36cd8c8203ad"} Nov 03 17:39:22.030209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.030174 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx"] Nov 03 17:39:22.033608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.033585 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba88695-q6tnx"] Nov 03 17:39:22.216749 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.216698 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podStartSLOduration=2.216681972 podStartE2EDuration="2.216681972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:22.06772149 +0000 UTC m=+5057.400935060" watchObservedRunningTime="2026-11-03 17:39:22.216681972 +0000 UTC m=+5057.549895538" Nov 03 17:39:22.218536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.218511 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8"] Nov 03 17:39:22.222138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.222112 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.261522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.258575 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:39:22.261522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.258866 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:39:22.261522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.259030 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:39:22.261522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.259322 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:39:22.265464 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.262259 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8"] Nov 03 17:39:22.279958 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:39:22.279915 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:39:22.323587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.323552 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.323768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.323614 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbn5\" (UniqueName: \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.323768 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.323744 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.424492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.424428 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbn5\" (UniqueName: \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.424693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.424530 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.424693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.424580 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.425255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.425229 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.426947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.426928 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.467471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.459207 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbn5\" (UniqueName: \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\") pod \"maas-api-revoke-sub-85237e45-8n5n8\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.536000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.535910 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:22.788429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.788345 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8e1aec0-61f1-4c17-9ded-60cc47c994a5" path="/var/lib/kubelet/pods/f8e1aec0-61f1-4c17-9ded-60cc47c994a5/volumes" Nov 03 17:39:22.871814 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.871782 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8"] Nov 03 17:39:22.872970 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:39:22.872938 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3faf336_2167_4339_9b6c_0fd69f411336.slice/crio-5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055 WatchSource:0}: Error finding container 5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055: Status 404 returned error can't find the container with id 5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055 Nov 03 17:39:22.969267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.963897 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:22.970635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.970575 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:22.970783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:22.970658 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:23.003804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:23.003760 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" event={"ID":"f3faf336-2167-4339-9b6c-0fd69f411336","Type":"ContainerStarted","Data":"5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055"} Nov 03 17:39:23.035817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:23.035782 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd"] Nov 03 17:39:23.062533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:23.039507 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac958a16-l45zd"] Nov 03 17:39:23.964466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:23.962935 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:23.964466 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:23.963019 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:24.011280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:24.011248 6001 generic.go:356] "Generic (PLEG): container finished" podID="f3faf336-2167-4339-9b6c-0fd69f411336" containerID="48bcf0a05e7749a33af592ae8ca959804274c1950d05699031e3cd28722c4ec7" exitCode=0 Nov 03 17:39:24.011757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:24.011542 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" event={"ID":"f3faf336-2167-4339-9b6c-0fd69f411336","Type":"ContainerDied","Data":"48bcf0a05e7749a33af592ae8ca959804274c1950d05699031e3cd28722c4ec7"} Nov 03 17:39:24.789158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:24.789125 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b5ef8a-8732-4972-ba05-75c931862789" path="/var/lib/kubelet/pods/28b5ef8a-8732-4972-ba05-75c931862789/volumes" Nov 03 17:39:24.964468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:24.963922 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:24.964468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:24.963987 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:25.199035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.199004 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:25.275965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.275931 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\") pod \"f3faf336-2167-4339-9b6c-0fd69f411336\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " Nov 03 17:39:25.276199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.276014 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\") pod \"f3faf336-2167-4339-9b6c-0fd69f411336\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " Nov 03 17:39:25.276199 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.276068 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\" (UniqueName: \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\") pod \"f3faf336-2167-4339-9b6c-0fd69f411336\" (UID: \"f3faf336-2167-4339-9b6c-0fd69f411336\") " Nov 03 17:39:25.276461 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.276418 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca" pod "f3faf336-2167-4339-9b6c-0fd69f411336" (UID: "f3faf336-2167-4339-9b6c-0fd69f411336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:39:25.278474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.278430 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5" pod "f3faf336-2167-4339-9b6c-0fd69f411336" (UID: "f3faf336-2167-4339-9b6c-0fd69f411336"). InnerVolumeSpecName "kube-api-access-pqbn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:25.278855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.278835 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca" pod "f3faf336-2167-4339-9b6c-0fd69f411336" (UID: "f3faf336-2167-4339-9b6c-0fd69f411336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:25.377190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.377097 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:39:25.377190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.377134 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3faf336-2167-4339-9b6c-0fd69f411336-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:39:25.377190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.377143 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbn5\" (UniqueName: \"kubernetes.io/projected/f3faf336-2167-4339-9b6c-0fd69f411336-kube-api-access-pqbn5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:39:25.968468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.961846 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:25.968468 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:25.961914 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:26.024189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:26.024161 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" Nov 03 17:39:26.024480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:26.024189 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8" event={"ID":"f3faf336-2167-4339-9b6c-0fd69f411336","Type":"ContainerDied","Data":"5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055"} Nov 03 17:39:26.024565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:26.024507 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cd1fd9cf06f6436dc3f4dbe1fac95c831c42371ac8b42c750fed72d10bc3055" Nov 03 17:39:26.964472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:26.962853 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:26.964472 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:26.962922 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:27.962351 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:27.962311 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" start-of-body= Nov 03 17:39:27.962565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:27.962373 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": dial tcp 10.134.1.176:15021: connect: connection refused" Nov 03 17:39:29.963567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:29.963530 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:29.964058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:29.963590 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:31.005607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:31.005562 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:31.006152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:31.005622 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:31.949035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:31.948997 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:31.949360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:31.949334 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:31.950185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:31.950166 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" Nov 03 17:39:33.029757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:33.029727 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf"] Nov 03 17:39:33.041952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:33.041925 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e59103c8-wmfpf"] Nov 03 17:39:34.037832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:34.037801 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf"] Nov 03 17:39:34.050242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:34.050214 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c954b259-68znf"] Nov 03 17:39:34.788408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:34.788364 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b" path="/var/lib/kubelet/pods/25a2be10-d0b6-4f83-bbc4-fc3d84a2a43b/volumes" Nov 03 17:39:34.788878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:34.788855 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea22dca3-e226-47d6-9954-2093b9ee3e80" path="/var/lib/kubelet/pods/ea22dca3-e226-47d6-9954-2093b9ee3e80/volumes" Nov 03 17:39:35.673887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.673843 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j"] Nov 03 17:39:35.674312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.674289 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f3faf336-2167-4339-9b6c-0fd69f411336" containerName="revoke-keys" Nov 03 17:39:35.674312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.674303 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3faf336-2167-4339-9b6c-0fd69f411336" containerName="revoke-keys" Nov 03 17:39:35.674401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.674387 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3faf336-2167-4339-9b6c-0fd69f411336" containerName="revoke-keys" Nov 03 17:39:35.678917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.678890 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.706871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.706831 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j"] Nov 03 17:39:35.787973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.787931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b971155-38f4-436e-b6a8-b7224b30286e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788350 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788422 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788469 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cq2g\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-kube-api-access-9cq2g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788506 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788591 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b971155-38f4-436e-b6a8-b7224b30286e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788643 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788676 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.788766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.788701 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.889866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.889820 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b971155-38f4-436e-b6a8-b7224b30286e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.889873 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.889896 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.889921 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.889974 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b971155-38f4-436e-b6a8-b7224b30286e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890009 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890056 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890077 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cq2g\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-kube-api-access-9cq2g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890112 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890395 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890636 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890717 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890722 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b971155-38f4-436e-b6a8-b7224b30286e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.890805 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.890786 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.892552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.892531 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b971155-38f4-436e-b6a8-b7224b30286e-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.892679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.892665 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b971155-38f4-436e-b6a8-b7224b30286e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.913883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.913851 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.914069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.914047 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cq2g\" (UniqueName: \"kubernetes.io/projected/7b971155-38f4-436e-b6a8-b7224b30286e-kube-api-access-9cq2g\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-xsc9j\" (UID: \"7b971155-38f4-436e-b6a8-b7224b30286e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:35.994287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:35.994261 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:36.312437 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:36.312406 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j"] Nov 03 17:39:36.314812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:36.314781 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:36.314916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:36.314845 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:36.314916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:36.314874 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:39:37.092659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.092614 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" event={"ID":"7b971155-38f4-436e-b6a8-b7224b30286e","Type":"ContainerStarted","Data":"a7c57361bd174ffc70d0fd493cd4cc919e18e727be6ad9e4a05bfe109a27305a"} Nov 03 17:39:37.092659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.092668 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" event={"ID":"7b971155-38f4-436e-b6a8-b7224b30286e","Type":"ContainerStarted","Data":"7b7b2afb0bde752a4dea085b855601e67be1a9c2db7e5fea8656985c9332d98e"} Nov 03 17:39:37.169009 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.168939 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podStartSLOduration=2.168922468 podStartE2EDuration="2.168922468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:37.167627971 +0000 UTC m=+5072.500841541" watchObservedRunningTime="2026-11-03 17:39:37.168922468 +0000 UTC m=+5072.502136040" Nov 03 17:39:37.994578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.994540 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:37.996113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.996087 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:37.996255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:37.996133 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:38.994837 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:38.994797 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:38.995266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:38.994860 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:39.994890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:39.994843 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:39.995484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:39.994909 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:40.995035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:40.994998 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:40.995495 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:40.995067 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:41.995590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:41.995544 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:41.996140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:41.995627 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:42.994949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:42.994922 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" start-of-body= Nov 03 17:39:42.995165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:42.994975 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": dial tcp 10.134.1.178:15021: connect: connection refused" Nov 03 17:39:44.661696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:44.661658 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:44.662162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:44.661989 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:44.663206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:39:44.663184 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" Nov 03 17:39:52.250560 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:39:52.250521 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:40:00.706643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:00.706587 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:40:00.707340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:00.706665 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:40:02.273063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.273022 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk"] Nov 03 17:40:02.276801 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.276773 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.292134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.292109 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:40:02.292134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.292108 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:40:02.304569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.304539 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:40:02.304569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.304541 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:40:02.306944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.306911 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk"] Nov 03 17:40:02.436261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.436216 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.436553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.436355 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.436553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.436487 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsb69\" (UniqueName: \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.538063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.537945 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsb69\" (UniqueName: \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.538282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.538065 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.538282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.538126 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.538963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.538943 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.540680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.540660 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.564297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.564257 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsb69\" (UniqueName: \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\") pod \"maas-api-revoke-sub-f904bff9-gp4pk\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.590953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.590911 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:02.898368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:02.898333 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk"] Nov 03 17:40:02.898971 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:40:02.898938 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7910922f_c874_4180_a440_76c4428d75b0.slice/crio-c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66 WatchSource:0}: Error finding container c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66: Status 404 returned error can't find the container with id c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66 Nov 03 17:40:03.255299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:03.255261 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" event={"ID":"7910922f-c874-4180-a440-76c4428d75b0","Type":"ContainerStarted","Data":"e4b98f130b1dbc9428bf3d7e1601a5c917bb14619630768f8745192cdd2c5584"} Nov 03 17:40:03.255557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:03.255318 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" event={"ID":"7910922f-c874-4180-a440-76c4428d75b0","Type":"ContainerStarted","Data":"c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66"} Nov 03 17:40:03.314835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:03.314785 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" podStartSLOduration=1.314771713 podStartE2EDuration="1.314771713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:03.313782214 +0000 UTC m=+5098.646995783" watchObservedRunningTime="2026-11-03 17:40:03.314771713 +0000 UTC m=+5098.647985281" Nov 03 17:40:04.262594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:04.262561 6001 generic.go:356] "Generic (PLEG): container finished" podID="7910922f-c874-4180-a440-76c4428d75b0" containerID="e4b98f130b1dbc9428bf3d7e1601a5c917bb14619630768f8745192cdd2c5584" exitCode=0 Nov 03 17:40:04.262798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:04.262634 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" event={"ID":"7910922f-c874-4180-a440-76c4428d75b0","Type":"ContainerDied","Data":"e4b98f130b1dbc9428bf3d7e1601a5c917bb14619630768f8745192cdd2c5584"} Nov 03 17:40:05.401558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.401530 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:05.569201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.569115 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\") pod \"7910922f-c874-4180-a440-76c4428d75b0\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " Nov 03 17:40:05.569201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.569179 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\") pod \"7910922f-c874-4180-a440-76c4428d75b0\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " Nov 03 17:40:05.569425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.569243 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\" (UniqueName: \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\") pod \"7910922f-c874-4180-a440-76c4428d75b0\" (UID: \"7910922f-c874-4180-a440-76c4428d75b0\") " Nov 03 17:40:05.569567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.569543 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca" pod "7910922f-c874-4180-a440-76c4428d75b0" (UID: "7910922f-c874-4180-a440-76c4428d75b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:40:05.569731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.569717 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7910922f-c874-4180-a440-76c4428d75b0-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:05.571349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.571327 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca" pod "7910922f-c874-4180-a440-76c4428d75b0" (UID: "7910922f-c874-4180-a440-76c4428d75b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:40:05.571487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.571401 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69" pod "7910922f-c874-4180-a440-76c4428d75b0" (UID: "7910922f-c874-4180-a440-76c4428d75b0"). InnerVolumeSpecName "kube-api-access-fsb69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:40:05.670922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.670890 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsb69\" (UniqueName: \"kubernetes.io/projected/7910922f-c874-4180-a440-76c4428d75b0-kube-api-access-fsb69\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:05.670922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.670916 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7910922f-c874-4180-a440-76c4428d75b0-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:05.860014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.859937 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:40:05.973079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:05.973050 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:40:06.274494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:06.274465 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" Nov 03 17:40:06.274494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:06.274476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk" event={"ID":"7910922f-c874-4180-a440-76c4428d75b0","Type":"ContainerDied","Data":"c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66"} Nov 03 17:40:06.274727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:06.274529 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96753e5d433553d624af76ee26ada0f068a2a624c0d95be4f149f01c446bd66" Nov 03 17:40:14.236649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:14.236612 6001 scope.go:122] "RemoveContainer" containerID="57a6d745077403fc8f222dd60b69de64e97955d2c70f6c2be7682212244c8e20" Nov 03 17:40:21.058928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:21.058886 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w"] Nov 03 17:40:21.071273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:21.071237 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fccf1837-msm7w"] Nov 03 17:40:22.251073 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:40:22.251031 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:40:22.788586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:22.788550 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc4e99f-ab2b-4144-a4e7-d8a97537f656" path="/var/lib/kubelet/pods/9fc4e99f-ab2b-4144-a4e7-d8a97537f656/volumes" Nov 03 17:40:26.759044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.758999 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs"] Nov 03 17:40:26.759701 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.759677 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7910922f-c874-4180-a440-76c4428d75b0" containerName="revoke-keys" Nov 03 17:40:26.759811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.759703 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7910922f-c874-4180-a440-76c4428d75b0" containerName="revoke-keys" Nov 03 17:40:26.759892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.759878 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7910922f-c874-4180-a440-76c4428d75b0" containerName="revoke-keys" Nov 03 17:40:26.762889 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.762866 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.779534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.779505 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:40:26.791931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.791899 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:40:26.792135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.791976 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:40:26.792616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.792588 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:40:26.796024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.795994 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs"] Nov 03 17:40:26.844695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.844656 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghb5\" (UniqueName: \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.844897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.844725 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.844897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.844853 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.946150 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.946111 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.946372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.946227 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghb5\" (UniqueName: \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.946372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.946288 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.947016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.946995 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.948605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.948578 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:26.973523 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:26.973493 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghb5\" (UniqueName: \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\") pod \"maas-api-revoke-sub-f03629db-6qzrs\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:27.075981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:27.075889 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:27.384319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:27.384286 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs"] Nov 03 17:40:27.385243 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:40:27.385217 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5260ea57_d51f_4b5c_90d3_58f52b68549b.slice/crio-57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10 WatchSource:0}: Error finding container 57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10: Status 404 returned error can't find the container with id 57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10 Nov 03 17:40:27.442978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:27.442942 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" event={"ID":"5260ea57-d51f-4b5c-90d3-58f52b68549b","Type":"ContainerStarted","Data":"57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10"} Nov 03 17:40:28.450320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:28.450279 6001 generic.go:356] "Generic (PLEG): container finished" podID="5260ea57-d51f-4b5c-90d3-58f52b68549b" containerID="f5dd9f067b701039f7f19c286e27601507ce0d45c494c6f710ab062cbe5123cc" exitCode=0 Nov 03 17:40:28.450804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:28.450375 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" event={"ID":"5260ea57-d51f-4b5c-90d3-58f52b68549b","Type":"ContainerDied","Data":"f5dd9f067b701039f7f19c286e27601507ce0d45c494c6f710ab062cbe5123cc"} Nov 03 17:40:29.615471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.615429 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:29.675298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.675260 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\" (UniqueName: \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\") pod \"5260ea57-d51f-4b5c-90d3-58f52b68549b\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " Nov 03 17:40:29.675531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.675493 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\") pod \"5260ea57-d51f-4b5c-90d3-58f52b68549b\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " Nov 03 17:40:29.675628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.675545 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\") pod \"5260ea57-d51f-4b5c-90d3-58f52b68549b\" (UID: \"5260ea57-d51f-4b5c-90d3-58f52b68549b\") " Nov 03 17:40:29.675982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.675954 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca" pod "5260ea57-d51f-4b5c-90d3-58f52b68549b" (UID: "5260ea57-d51f-4b5c-90d3-58f52b68549b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:40:29.677693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.677665 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5" pod "5260ea57-d51f-4b5c-90d3-58f52b68549b" (UID: "5260ea57-d51f-4b5c-90d3-58f52b68549b"). InnerVolumeSpecName "kube-api-access-bghb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:40:29.677786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.677726 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca" pod "5260ea57-d51f-4b5c-90d3-58f52b68549b" (UID: "5260ea57-d51f-4b5c-90d3-58f52b68549b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:40:29.777222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.777187 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:29.777222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.777218 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5260ea57-d51f-4b5c-90d3-58f52b68549b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:29.777222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:29.777231 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghb5\" (UniqueName: \"kubernetes.io/projected/5260ea57-d51f-4b5c-90d3-58f52b68549b-kube-api-access-bghb5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:30.463998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:30.463966 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" Nov 03 17:40:30.464235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:30.464000 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs" event={"ID":"5260ea57-d51f-4b5c-90d3-58f52b68549b","Type":"ContainerDied","Data":"57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10"} Nov 03 17:40:30.464235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:30.464050 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57a9d0f1528f187e58fbddaf8f8b64ed34c7c82d313f5fd6c8076e445b410e10" Nov 03 17:40:32.031242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:32.031200 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6"] Nov 03 17:40:32.043391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:32.043356 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-760c3229-xbjg6"] Nov 03 17:40:32.787786 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:32.787748 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319fc1ea-8e21-4421-9e32-b7e0d099b54c" path="/var/lib/kubelet/pods/319fc1ea-8e21-4421-9e32-b7e0d099b54c/volumes" Nov 03 17:40:42.030053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:42.030010 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg"] Nov 03 17:40:42.042782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:42.042742 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b998cb-nf8mg"] Nov 03 17:40:42.789543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:42.789495 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed49247f-bf6d-45e8-bc86-cd474c1a50ed" path="/var/lib/kubelet/pods/ed49247f-bf6d-45e8-bc86-cd474c1a50ed/volumes" Nov 03 17:40:43.012550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.012519 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp"] Nov 03 17:40:43.012987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.012974 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5260ea57-d51f-4b5c-90d3-58f52b68549b" containerName="revoke-keys" Nov 03 17:40:43.013041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.012988 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5260ea57-d51f-4b5c-90d3-58f52b68549b" containerName="revoke-keys" Nov 03 17:40:43.013089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.013061 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5260ea57-d51f-4b5c-90d3-58f52b68549b" containerName="revoke-keys" Nov 03 17:40:43.017661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.017637 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.040900 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.040816 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:40:43.046808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.046771 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp"] Nov 03 17:40:43.053545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.053506 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:40:43.053545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.053530 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:40:43.053772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.053630 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:40:43.125146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.125104 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.125348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.125286 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.125348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.125328 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzxx\" (UniqueName: \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.226861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.226807 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.227082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.226878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzxx\" (UniqueName: \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.227082 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.226951 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.227639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.227613 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.229439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.229414 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.251180 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.251140 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzxx\" (UniqueName: \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\") pod \"maas-api-revoke-sub-0165d7d2-v98sp\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.332118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.332006 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:43.636607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:43.636569 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp"] Nov 03 17:40:43.637884 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:40:43.637856 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5890fb3_d052_458a_ad8e_dab457674fc5.slice/crio-69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279 WatchSource:0}: Error finding container 69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279: Status 404 returned error can't find the container with id 69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279 Nov 03 17:40:44.549998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:44.549955 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" event={"ID":"a5890fb3-d052-458a-ad8e-dab457674fc5","Type":"ContainerStarted","Data":"3c06c9a1f1e5960851200c0a23523659ebecb5a9a9391fad66123c43e9d62bb0"} Nov 03 17:40:44.550412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:44.550034 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" event={"ID":"a5890fb3-d052-458a-ad8e-dab457674fc5","Type":"ContainerStarted","Data":"69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279"} Nov 03 17:40:44.593023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:44.592956 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" podStartSLOduration=2.592940761 podStartE2EDuration="2.592940761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:44.591729617 +0000 UTC m=+5139.924943180" watchObservedRunningTime="2026-11-03 17:40:44.592940761 +0000 UTC m=+5139.926154330" Nov 03 17:40:45.558365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:45.558333 6001 generic.go:356] "Generic (PLEG): container finished" podID="a5890fb3-d052-458a-ad8e-dab457674fc5" containerID="3c06c9a1f1e5960851200c0a23523659ebecb5a9a9391fad66123c43e9d62bb0" exitCode=0 Nov 03 17:40:45.558827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:45.558429 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" event={"ID":"a5890fb3-d052-458a-ad8e-dab457674fc5","Type":"ContainerDied","Data":"3c06c9a1f1e5960851200c0a23523659ebecb5a9a9391fad66123c43e9d62bb0"} Nov 03 17:40:46.722117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.722085 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:46.773841 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.773805 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\") pod \"a5890fb3-d052-458a-ad8e-dab457674fc5\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " Nov 03 17:40:46.774052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.773985 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\" (UniqueName: \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\") pod \"a5890fb3-d052-458a-ad8e-dab457674fc5\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " Nov 03 17:40:46.774052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.774032 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\") pod \"a5890fb3-d052-458a-ad8e-dab457674fc5\" (UID: \"a5890fb3-d052-458a-ad8e-dab457674fc5\") " Nov 03 17:40:46.774512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.774239 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca" pod "a5890fb3-d052-458a-ad8e-dab457674fc5" (UID: "a5890fb3-d052-458a-ad8e-dab457674fc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:40:46.774512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.774388 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:46.776418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.776389 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx" pod "a5890fb3-d052-458a-ad8e-dab457674fc5" (UID: "a5890fb3-d052-458a-ad8e-dab457674fc5"). InnerVolumeSpecName "kube-api-access-blzxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:40:46.776797 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.776771 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca" pod "a5890fb3-d052-458a-ad8e-dab457674fc5" (UID: "a5890fb3-d052-458a-ad8e-dab457674fc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:40:46.875112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.875025 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blzxx\" (UniqueName: \"kubernetes.io/projected/a5890fb3-d052-458a-ad8e-dab457674fc5-kube-api-access-blzxx\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:46.875112 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:46.875056 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5890fb3-d052-458a-ad8e-dab457674fc5-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:47.582618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:47.582573 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" Nov 03 17:40:47.582618 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:47.582598 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp" event={"ID":"a5890fb3-d052-458a-ad8e-dab457674fc5","Type":"ContainerDied","Data":"69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279"} Nov 03 17:40:47.582961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:47.582656 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f293fe7f5d5c6dfd873679c8c2f1966d663af237be5ab7fd4eb9452e0c5279" Nov 03 17:40:52.250920 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:40:52.250881 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:40:53.031769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:53.031733 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b"] Nov 03 17:40:53.045250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:53.045212 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ec88cbb-59w5b"] Nov 03 17:40:54.005340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.005299 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv"] Nov 03 17:40:54.005788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.005737 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a5890fb3-d052-458a-ad8e-dab457674fc5" containerName="revoke-keys" Nov 03 17:40:54.005788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.005748 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5890fb3-d052-458a-ad8e-dab457674fc5" containerName="revoke-keys" Nov 03 17:40:54.005886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.005846 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5890fb3-d052-458a-ad8e-dab457674fc5" containerName="revoke-keys" Nov 03 17:40:54.009055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.009033 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.024719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.024694 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:40:54.024854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.024737 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:40:54.032137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.032109 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv"] Nov 03 17:40:54.037811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.037787 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:40:54.037811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.037791 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:40:54.049947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.049921 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.050084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.049968 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6clj\" (UniqueName: \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.050200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.050175 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.151593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.151549 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.151798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.151606 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6clj\" (UniqueName: \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.151798 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.151662 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.152322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.152301 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.153958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.153931 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.176061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.176029 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6clj\" (UniqueName: \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\") pod \"maas-api-revoke-sub-6b19338f-nvspv\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.322165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.322078 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:54.630420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.630386 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv"] Nov 03 17:40:54.631168 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:40:54.631136 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ebb6c6_c821_4c18_aa7f_f03f8ed13595.slice/crio-25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3 WatchSource:0}: Error finding container 25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3: Status 404 returned error can't find the container with id 25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3 Nov 03 17:40:54.789163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:54.789128 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304852bb-c286-4b2f-b407-3eb5516072b6" path="/var/lib/kubelet/pods/304852bb-c286-4b2f-b407-3eb5516072b6/volumes" Nov 03 17:40:55.629393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:55.629356 6001 generic.go:356] "Generic (PLEG): container finished" podID="e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" containerID="43a682d238a3092829d429d6778fd2af729851ee756b6ec3d15653473176cec8" exitCode=0 Nov 03 17:40:55.629843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:55.629434 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" event={"ID":"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595","Type":"ContainerDied","Data":"43a682d238a3092829d429d6778fd2af729851ee756b6ec3d15653473176cec8"} Nov 03 17:40:55.629843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:55.629513 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" event={"ID":"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595","Type":"ContainerStarted","Data":"25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3"} Nov 03 17:40:56.774851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.774821 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:56.880919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.880890 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\") pod \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " Nov 03 17:40:56.881096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.881007 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\" (UniqueName: \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\") pod \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " Nov 03 17:40:56.881096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.881048 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\") pod \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\" (UID: \"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595\") " Nov 03 17:40:56.881321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.881292 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca" pod "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" (UID: "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:40:56.883232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.883201 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca" pod "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" (UID: "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:40:56.883361 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.883243 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj" pod "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" (UID: "e7ebb6c6-c821-4c18-aa7f-f03f8ed13595"). InnerVolumeSpecName "kube-api-access-l6clj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:40:56.982532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.982493 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6clj\" (UniqueName: \"kubernetes.io/projected/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-kube-api-access-l6clj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:56.982532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.982526 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:56.982532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:56.982537 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:40:57.642600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:57.642565 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" event={"ID":"e7ebb6c6-c821-4c18-aa7f-f03f8ed13595","Type":"ContainerDied","Data":"25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3"} Nov 03 17:40:57.642600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:57.642596 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv" Nov 03 17:40:57.642858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:40:57.642626 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25b24706be6448a36729a9f13d18e14c9bc9d86cfcc19d4e6a339562c4c939c3" Nov 03 17:41:07.306510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.306456 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:41:07.307220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.307171 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" containerName="revoke-keys" Nov 03 17:41:07.307220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.307195 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" containerName="revoke-keys" Nov 03 17:41:07.307381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.307339 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" containerName="revoke-keys" Nov 03 17:41:07.317325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.317294 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.337140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.337102 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:41:07.349824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.349785 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-39e7fadb-kserve-self-signed-certs\"" Nov 03 17:41:07.350221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.350155 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-39e7fadb-epp-sa-dockercfg-dh9s2\"" Nov 03 17:41:07.432657 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.432611 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.432898 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.432676 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trz7b\" (UniqueName: \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.533863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.533814 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.533863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.533870 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trz7b\" (UniqueName: \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.536716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.536677 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.565656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.565527 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trz7b\" (UniqueName: \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\") pod \"e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.643268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.643228 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:07.961064 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.961034 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:41:07.961938 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:41:07.961909 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb2010b_3bed_45a8_b394_e3967b90614b.slice/crio-df16df6188869f255a34bc0b29e4a4463935c51ea9dc9b304c6567a97560a0ba WatchSource:0}: Error finding container df16df6188869f255a34bc0b29e4a4463935c51ea9dc9b304c6567a97560a0ba: Status 404 returned error can't find the container with id df16df6188869f255a34bc0b29e4a4463935c51ea9dc9b304c6567a97560a0ba Nov 03 17:41:07.964937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:07.964914 6001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:41:08.741563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:08.741520 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" event={"ID":"fbb2010b-3bed-45a8-b394-e3967b90614b","Type":"ContainerStarted","Data":"df16df6188869f255a34bc0b29e4a4463935c51ea9dc9b304c6567a97560a0ba"} Nov 03 17:41:09.751436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:09.751389 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" event={"ID":"fbb2010b-3bed-45a8-b394-e3967b90614b","Type":"ContainerStarted","Data":"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8"} Nov 03 17:41:09.752013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:09.751655 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:09.798230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:09.798171 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podStartSLOduration=1.30330772 podStartE2EDuration="2.798153861s" totalImagesPullingTime="1.494846141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:09.797713008 +0000 UTC m=+5165.130926581" watchObservedRunningTime="2026-11-03 17:41:09.798153861 +0000 UTC m=+5165.131367420" Nov 03 17:41:14.261178 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:14.261145 6001 scope.go:122] "RemoveContainer" containerID="c92489003103a00bf2a2d7b4197fc8a16d67da9859f6825839b14f7d79b426e6" Nov 03 17:41:14.270208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:14.270186 6001 scope.go:122] "RemoveContainer" containerID="d11f0008a2f4308cdac83b6437de5b189544cba1a37173593ec3c009810038e7" Nov 03 17:41:14.278827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:14.278808 6001 scope.go:122] "RemoveContainer" containerID="3581f200863f94a8711afb5bd080ce454973b4ce5cb7bc43704ff760d48e8a92" Nov 03 17:41:14.287088 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:14.287070 6001 scope.go:122] "RemoveContainer" containerID="6593cc0ee3921b99160b46961034c79b353edd6531841f82a28c7fcb929231de" Nov 03 17:41:21.030475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:21.030413 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85"] Nov 03 17:41:21.043077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:21.043042 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84906a43-htz85"] Nov 03 17:41:22.251095 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:41:22.251056 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:41:22.788584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:22.788553 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5" path="/var/lib/kubelet/pods/b9ebc72a-748b-4fbe-8e2e-b11a4b766ef5/volumes" Nov 03 17:41:32.029135 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:32.029094 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7"] Nov 03 17:41:32.041524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:32.041496 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dd760d1-l5wd7"] Nov 03 17:41:32.789019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:32.788983 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad26f876-121c-4472-94d9-5e13411515cf" path="/var/lib/kubelet/pods/ad26f876-121c-4472-94d9-5e13411515cf/volumes" Nov 03 17:41:40.763951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:40.763919 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:41:41.030298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:41.030217 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx"] Nov 03 17:41:41.031917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:41.031894 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0213989f-klxmx"] Nov 03 17:41:42.788458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:42.788411 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af5c1398-18fb-474c-8a83-ead621920c9c" path="/var/lib/kubelet/pods/af5c1398-18fb-474c-8a83-ead621920c9c/volumes" Nov 03 17:41:52.323650 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:41:52.323602 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:41:53.036186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:53.036143 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb"] Nov 03 17:41:53.046587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:53.046554 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bc362d1-dclbb"] Nov 03 17:41:54.788075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:41:54.788042 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161a8cb9-a40d-4ea0-a776-6a07ecc3cb06" path="/var/lib/kubelet/pods/161a8cb9-a40d-4ea0-a776-6a07ecc3cb06/volumes" Nov 03 17:42:08.951996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:08.951956 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:42:10.763969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:10.763927 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:14.322882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:14.322843 6001 scope.go:122] "RemoveContainer" containerID="a49f1c42a45304d58ac0863e5fa2f957df24738128ca8db4eb8835d4b11b7bbc" Nov 03 17:42:14.332862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:14.332841 6001 scope.go:122] "RemoveContainer" containerID="dad93b033fac8c14e2478480ed026d8f21c967dea76fbe85384bd1ae09d4cde7" Nov 03 17:42:14.341408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:14.341387 6001 scope.go:122] "RemoveContainer" containerID="c621f5557618da1cb95662436df79fe303403804c364164af2331ac1bf9efa94" Nov 03 17:42:14.412629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:14.412603 6001 scope.go:122] "RemoveContainer" containerID="5de2c2aab1bc652684f79af5e267051f39d244534f9ff85b6fbc3a9a65f1fd5c" Nov 03 17:42:20.793538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:20.788558 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:22.249789 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:42:22.249751 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:42:23.996340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:23.996299 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" containerID="cri-o://3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8" gracePeriod=45 Nov 03 17:42:29.031639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:29.031598 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf"] Nov 03 17:42:29.044091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:29.044058 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcb3e408-vf2xf"] Nov 03 17:42:30.763915 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:30.763863 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:30.764378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:30.763973 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:42:30.788210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:30.788174 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b6b0804-b261-449d-a4e3-d40e34f1ce49" path="/var/lib/kubelet/pods/1b6b0804-b261-449d-a4e3-d40e34f1ce49/volumes" Nov 03 17:42:40.764312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:40.764270 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:47.038693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:47.038648 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d"] Nov 03 17:42:47.050880 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:47.050846 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3609fe8-dqx4d"] Nov 03 17:42:48.800204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:48.799774 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded59f61-f739-4a97-be3f-8cd30370db6c" path="/var/lib/kubelet/pods/ded59f61-f739-4a97-be3f-8cd30370db6c/volumes" Nov 03 17:42:50.763953 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:50.763912 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:42:52.250804 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:42:52.250760 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:42:54.278496 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.278469 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:42:54.389408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.389365 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\" (UniqueName: \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\") pod \"fbb2010b-3bed-45a8-b394-e3967b90614b\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " Nov 03 17:42:54.389636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.389524 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\") pod \"fbb2010b-3bed-45a8-b394-e3967b90614b\" (UID: \"fbb2010b-3bed-45a8-b394-e3967b90614b\") " Nov 03 17:42:54.391819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.391783 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b" pod "fbb2010b-3bed-45a8-b394-e3967b90614b" (UID: "fbb2010b-3bed-45a8-b394-e3967b90614b"). InnerVolumeSpecName "kube-api-access-trz7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:54.391948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.391826 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs" pod "fbb2010b-3bed-45a8-b394-e3967b90614b" (UID: "fbb2010b-3bed-45a8-b394-e3967b90614b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:54.428574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.428541 6001 generic.go:356] "Generic (PLEG): container finished" podID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerID="3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8" exitCode=0 Nov 03 17:42:54.428782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.428586 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" Nov 03 17:42:54.428782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.428634 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" event={"ID":"fbb2010b-3bed-45a8-b394-e3967b90614b","Type":"ContainerDied","Data":"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8"} Nov 03 17:42:54.428782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.428698 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv" event={"ID":"fbb2010b-3bed-45a8-b394-e3967b90614b","Type":"ContainerDied","Data":"df16df6188869f255a34bc0b29e4a4463935c51ea9dc9b304c6567a97560a0ba"} Nov 03 17:42:54.428782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.428729 6001 scope.go:122] "RemoveContainer" containerID="3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8" Nov 03 17:42:54.438838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.438812 6001 scope.go:122] "RemoveContainer" containerID="3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8" Nov 03 17:42:54.439175 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:42:54.439154 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8\": container with ID starting with 3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8 not found: ID does not exist" containerID="3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8" Nov 03 17:42:54.439236 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.439187 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8"} err="failed to get container status \"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8\": rpc error: code = NotFound desc = could not find container \"3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8\": container with ID starting with 3b8d8c49d31f313ead4fd99d146297a4a5c681dde8c08db92c218d1ddda6eaa8 not found: ID does not exist" Nov 03 17:42:54.481017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.480978 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:42:54.486995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.486964 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-39e7fadb-kserve-router-scheduler-df64745f-lxlmv"] Nov 03 17:42:54.490712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.490683 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trz7b\" (UniqueName: \"kubernetes.io/projected/fbb2010b-3bed-45a8-b394-e3967b90614b-kube-api-access-trz7b\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:42:54.490815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.490714 6001 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbb2010b-3bed-45a8-b394-e3967b90614b-tls-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:42:54.787839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:42:54.787805 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" path="/var/lib/kubelet/pods/fbb2010b-3bed-45a8-b394-e3967b90614b/volumes" Nov 03 17:43:14.500283 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:14.500246 6001 scope.go:122] "RemoveContainer" containerID="93c4b61bdbc3272ced84f3992df48242ef8e414644b167f8e6e0aff81befa981" Nov 03 17:43:14.509735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:14.509708 6001 scope.go:122] "RemoveContainer" containerID="9655790f31832a7dc2cd8c809691c0ee170cb59d275b36f3c7c5a8626011ec3a" Nov 03 17:43:14.518362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:14.518336 6001 scope.go:122] "RemoveContainer" containerID="e085e2c51dd73cadbbe6a350b0a278f80cfa03cd334cbc7d824c05d3fd86ae70" Nov 03 17:43:14.527200 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:14.527176 6001 scope.go:122] "RemoveContainer" containerID="6af24d009381ddec2e87967755756e5e4a196f7a3fdf3517c5e7826883237415" Nov 03 17:43:15.259223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.259181 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2"] Nov 03 17:43:15.259760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.259739 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" Nov 03 17:43:15.259886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.259761 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" Nov 03 17:43:15.259958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.259932 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb2010b-3bed-45a8-b394-e3967b90614b" containerName="main" Nov 03 17:43:15.265092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.265066 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.295560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.295516 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:43:15.295560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.295522 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:43:15.295560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.295548 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:43:15.295848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.295585 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:43:15.297536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.297506 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2"] Nov 03 17:43:15.409887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.409846 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczb5\" (UniqueName: \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.409887 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.409903 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.410204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.410122 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.510774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.510695 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.510774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.510746 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczb5\" (UniqueName: \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.511203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.510773 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.511389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.511366 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.513015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.512996 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.534673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.534648 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczb5\" (UniqueName: \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\") pod \"maas-api-revoke-sub-b73efcce-9nmc2\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.577529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.577489 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:15.892288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:15.892252 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2"] Nov 03 17:43:15.893486 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:43:15.893454 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d96a8f_873e_4e99_8a0c_80eaf4b6234f.slice/crio-c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f WatchSource:0}: Error finding container c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f: Status 404 returned error can't find the container with id c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f Nov 03 17:43:16.562114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:16.562071 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" event={"ID":"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f","Type":"ContainerStarted","Data":"2a0a3c5885313cac804a0c120f8d4a2a0e2a416c44a57679c5babb5ccf2be2b1"} Nov 03 17:43:16.562561 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:16.562144 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" event={"ID":"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f","Type":"ContainerStarted","Data":"c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f"} Nov 03 17:43:16.608652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:16.608589 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" podStartSLOduration=1.608572909 podStartE2EDuration="1.608572909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:16.607402299 +0000 UTC m=+5291.940615868" watchObservedRunningTime="2026-11-03 17:43:16.608572909 +0000 UTC m=+5291.941786477" Nov 03 17:43:17.570256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:17.570215 6001 generic.go:356] "Generic (PLEG): container finished" podID="a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" containerID="2a0a3c5885313cac804a0c120f8d4a2a0e2a416c44a57679c5babb5ccf2be2b1" exitCode=0 Nov 03 17:43:17.570825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:17.570299 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" event={"ID":"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f","Type":"ContainerDied","Data":"2a0a3c5885313cac804a0c120f8d4a2a0e2a416c44a57679c5babb5ccf2be2b1"} Nov 03 17:43:18.717341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.717307 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:18.844428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.844324 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\") pod \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " Nov 03 17:43:18.844428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.844397 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\" (UniqueName: \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\") pod \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " Nov 03 17:43:18.844741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.844501 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\") pod \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\" (UID: \"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f\") " Nov 03 17:43:18.844923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.844894 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca" pod "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" (UID: "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:43:18.847172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.847144 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca" pod "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" (UID: "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:18.847172 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.847153 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5" pod "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" (UID: "a3d96a8f-873e-4e99-8a0c-80eaf4b6234f"). InnerVolumeSpecName "kube-api-access-mczb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:18.945806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.945764 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:43:18.945806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.945799 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:43:18.945806 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:18.945810 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczb5\" (UniqueName: \"kubernetes.io/projected/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f-kube-api-access-mczb5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:43:19.582373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:19.582342 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" Nov 03 17:43:19.582592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:19.582342 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2" event={"ID":"a3d96a8f-873e-4e99-8a0c-80eaf4b6234f","Type":"ContainerDied","Data":"c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f"} Nov 03 17:43:19.582592 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:43:19.582497 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1ddbcd1779a46cf79fe795a5d2f23cfe5ce4f0a691744c0328d58c4e27dca4f" Nov 03 17:43:22.252464 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:43:22.252405 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:43:52.251555 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:43:52.251464 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:44:05.876370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.876335 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:05.876868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.876843 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" containerName="revoke-keys" Nov 03 17:44:05.876868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.876861 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" containerName="revoke-keys" Nov 03 17:44:05.877028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.877012 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" containerName="revoke-keys" Nov 03 17:44:05.880424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.880409 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:05.896190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.896169 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4b1292\"" Nov 03 17:44:05.910168 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.910133 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:05.932718 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.932676 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:05.932913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.932754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c27mt\" (UniqueName: \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:05.937726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.937693 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:05.941646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.941622 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:05.956751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.956721 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-56341937-dockercfg-25xj9\"" Nov 03 17:44:05.963679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:05.963649 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:06.034173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.034128 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c27mt\" (UniqueName: \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.034465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.034294 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.034465 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.034426 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fc4\" (UniqueName: \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.034676 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.034528 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.035653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.035630 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.058606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.058564 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c27mt\" (UniqueName: \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.135671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.135565 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.135671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.135620 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fc4\" (UniqueName: \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.136377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.136349 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.159818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.159785 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fc4\" (UniqueName: \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.194626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.194585 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.257343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.255790 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.499134 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.499106 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:06.499924 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:44:06.499890 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cc0ad0_31f7_4342_9ddc_51ebeb2eac54.slice/crio-8e9068c54d61e87e22b3d52ddd71414672aa03dab05c6959153580e3dc138458 WatchSource:0}: Error finding container 8e9068c54d61e87e22b3d52ddd71414672aa03dab05c6959153580e3dc138458: Status 404 returned error can't find the container with id 8e9068c54d61e87e22b3d52ddd71414672aa03dab05c6959153580e3dc138458 Nov 03 17:44:06.567691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.567667 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:06.568048 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:44:06.568023 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e577a64_c6c3_4b27_a19d_315e172a872f.slice/crio-0cb799aaa331ca1085e4b5f46df471214269591934f5272b2c34e29017351c8f WatchSource:0}: Error finding container 0cb799aaa331ca1085e4b5f46df471214269591934f5272b2c34e29017351c8f: Status 404 returned error can't find the container with id 0cb799aaa331ca1085e4b5f46df471214269591934f5272b2c34e29017351c8f Nov 03 17:44:06.836016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.835918 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" event={"ID":"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54","Type":"ContainerStarted","Data":"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3"} Nov 03 17:44:06.836016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.835981 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" event={"ID":"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54","Type":"ContainerStarted","Data":"8e9068c54d61e87e22b3d52ddd71414672aa03dab05c6959153580e3dc138458"} Nov 03 17:44:06.836287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.836026 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:06.837664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.837637 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" event={"ID":"4e577a64-c6c3-4b27-a19d-315e172a872f","Type":"ContainerStarted","Data":"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f"} Nov 03 17:44:06.837810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.837683 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" event={"ID":"4e577a64-c6c3-4b27-a19d-315e172a872f","Type":"ContainerStarted","Data":"0cb799aaa331ca1085e4b5f46df471214269591934f5272b2c34e29017351c8f"} Nov 03 17:44:06.837810 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.837714 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:06.887789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.887728 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" podStartSLOduration=1.887715033 podStartE2EDuration="1.887715033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:06.887538201 +0000 UTC m=+5342.220751772" watchObservedRunningTime="2026-11-03 17:44:06.887715033 +0000 UTC m=+5342.220928601" Nov 03 17:44:06.930208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:06.930147 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" podStartSLOduration=1.930131421 podStartE2EDuration="1.930131421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:06.929718059 +0000 UTC m=+5342.262931644" watchObservedRunningTime="2026-11-03 17:44:06.930131421 +0000 UTC m=+5342.263344990" Nov 03 17:44:14.484475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.484421 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq"] Nov 03 17:44:14.488998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.488970 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.504638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.504613 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:44:14.516851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.516813 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:44:14.516851 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.516823 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:44:14.517085 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.516965 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:44:14.520096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.520072 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq"] Nov 03 17:44:14.566525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.566491 6001 scope.go:122] "RemoveContainer" containerID="e74b5b9e776e166505268bfb74d849f0b84736e8d8274c1e484e71695c6665f6" Nov 03 17:44:14.576870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.576845 6001 scope.go:122] "RemoveContainer" containerID="8a9070598e27f181d784dc99ceda2bd7ba11d889bbea883dde5eba6a7a09762e" Nov 03 17:44:14.622802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.622766 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.622980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.622806 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.622980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.622941 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg788\" (UniqueName: \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.724518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.724475 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg788\" (UniqueName: \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.724714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.724611 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.724714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.724636 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.725390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.725371 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.727123 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.727102 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.748046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.747973 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg788\" (UniqueName: \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\") pod \"maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:14.803670 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:14.803637 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:15.108899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:15.108864 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq"] Nov 03 17:44:15.109252 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:44:15.109226 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d63b08_f6a6_44ca_aa43_2514a1e07a9f.slice/crio-7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14 WatchSource:0}: Error finding container 7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14: Status 404 returned error can't find the container with id 7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14 Nov 03 17:44:15.891585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:15.891543 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" event={"ID":"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f","Type":"ContainerStarted","Data":"243324e12297a09cb01b151558daa74f49edc1c1ff89c784855fb2627d689e0a"} Nov 03 17:44:15.892028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:15.891611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" event={"ID":"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f","Type":"ContainerStarted","Data":"7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14"} Nov 03 17:44:15.933758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:15.933699 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" podStartSLOduration=1.933685651 podStartE2EDuration="1.933685651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:15.933183686 +0000 UTC m=+5351.266397256" watchObservedRunningTime="2026-11-03 17:44:15.933685651 +0000 UTC m=+5351.266899219" Nov 03 17:44:16.899151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:16.899117 6001 generic.go:356] "Generic (PLEG): container finished" podID="c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" containerID="243324e12297a09cb01b151558daa74f49edc1c1ff89c784855fb2627d689e0a" exitCode=0 Nov 03 17:44:16.899607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:16.899207 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" event={"ID":"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f","Type":"ContainerDied","Data":"243324e12297a09cb01b151558daa74f49edc1c1ff89c784855fb2627d689e0a"} Nov 03 17:44:17.845835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:17.845805 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:17.846052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:17.845873 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:18.066571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.066543 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:18.163126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.163020 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\") pod \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " Nov 03 17:44:18.163126 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.163098 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\") pod \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " Nov 03 17:44:18.163408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.163217 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\" (UniqueName: \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\") pod \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\" (UID: \"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f\") " Nov 03 17:44:18.163507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.163485 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca" pod "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" (UID: "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:44:18.163629 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.163612 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:18.165393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.165368 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788" pod "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" (UID: "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f"). InnerVolumeSpecName "kube-api-access-xg788". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:18.165393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.165377 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca" pod "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" (UID: "c2d63b08-f6a6-44ca-aa43-2514a1e07a9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:44:18.265138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.265099 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:18.265138 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.265131 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg788\" (UniqueName: \"kubernetes.io/projected/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f-kube-api-access-xg788\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:18.919804 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.919770 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" Nov 03 17:44:18.919991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.919767 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq" event={"ID":"c2d63b08-f6a6-44ca-aa43-2514a1e07a9f","Type":"ContainerDied","Data":"7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14"} Nov 03 17:44:18.919991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:18.919914 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc51cecc3025030ecdaccf9c05e8c071ddf22c7b47e0993a3a5d14e4b9bef14" Nov 03 17:44:22.249599 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:44:22.249563 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:44:24.908975 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:24.908934 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:24.941344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:24.941293 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:26.028114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:26.028076 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8"] Nov 03 17:44:26.040644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:26.040612 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-8n5n8"] Nov 03 17:44:26.789254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:26.789218 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3faf336-2167-4339-9b6c-0fd69f411336" path="/var/lib/kubelet/pods/f3faf336-2167-4339-9b6c-0fd69f411336/volumes" Nov 03 17:44:27.462783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:27.462743 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq"] Nov 03 17:44:27.475113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:27.475077 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-4b1292-bppgq"] Nov 03 17:44:28.788635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:28.788601 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" path="/var/lib/kubelet/pods/c2d63b08-f6a6-44ca-aa43-2514a1e07a9f/volumes" Nov 03 17:44:29.910090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:29.910000 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" podUID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" containerName="payload-processing" containerID="cri-o://4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3" gracePeriod=25 Nov 03 17:44:29.943204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:29.943166 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" podUID="4e577a64-c6c3-4b27-a19d-315e172a872f" containerName="payload-pre-processing" containerID="cri-o://78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f" gracePeriod=25 Nov 03 17:44:30.221808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.221738 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:30.250023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.249995 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:30.393184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393151 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\") pod \"4e577a64-c6c3-4b27-a19d-315e172a872f\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " Nov 03 17:44:30.393393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393205 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\" (UniqueName: \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\") pod \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " Nov 03 17:44:30.393393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393229 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\") pod \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\" (UID: \"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54\") " Nov 03 17:44:30.393393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393314 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\" (UniqueName: \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\") pod \"4e577a64-c6c3-4b27-a19d-315e172a872f\" (UID: \"4e577a64-c6c3-4b27-a19d-315e172a872f\") " Nov 03 17:44:30.393679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393657 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume" pod "e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" (UID: "e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:44:30.393728 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393684 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume" pod "4e577a64-c6c3-4b27-a19d-315e172a872f" (UID: "4e577a64-c6c3-4b27-a19d-315e172a872f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:44:30.393775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.393749 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:30.395741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.395719 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt" pod "e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" (UID: "e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54"). InnerVolumeSpecName "kube-api-access-c27mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:30.395832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.395749 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4" pod "4e577a64-c6c3-4b27-a19d-315e172a872f" (UID: "4e577a64-c6c3-4b27-a19d-315e172a872f"). InnerVolumeSpecName "kube-api-access-g8fc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:30.494386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.494355 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e577a64-c6c3-4b27-a19d-315e172a872f-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:30.494386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.494382 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c27mt\" (UniqueName: \"kubernetes.io/projected/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54-kube-api-access-c27mt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:30.494386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.494392 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fc4\" (UniqueName: \"kubernetes.io/projected/4e577a64-c6c3-4b27-a19d-315e172a872f-kube-api-access-g8fc4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:44:30.831967 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:44:30.831916 6001 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e577a64_c6c3_4b27_a19d_315e172a872f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cc0ad0_31f7_4342_9ddc_51ebeb2eac54.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:44:30.996019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.995973 6001 generic.go:356] "Generic (PLEG): container finished" podID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" containerID="4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3" exitCode=0 Nov 03 17:44:30.996571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.996043 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" Nov 03 17:44:30.996571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.996048 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" event={"ID":"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54","Type":"ContainerDied","Data":"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3"} Nov 03 17:44:30.996571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.996132 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r" event={"ID":"e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54","Type":"ContainerDied","Data":"8e9068c54d61e87e22b3d52ddd71414672aa03dab05c6959153580e3dc138458"} Nov 03 17:44:30.996571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.996154 6001 scope.go:122] "RemoveContainer" containerID="4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3" Nov 03 17:44:30.997839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.997802 6001 generic.go:356] "Generic (PLEG): container finished" podID="4e577a64-c6c3-4b27-a19d-315e172a872f" containerID="78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f" exitCode=0 Nov 03 17:44:30.997948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.997850 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" Nov 03 17:44:30.997948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.997883 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" event={"ID":"4e577a64-c6c3-4b27-a19d-315e172a872f","Type":"ContainerDied","Data":"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f"} Nov 03 17:44:30.998055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:30.997947 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq" event={"ID":"4e577a64-c6c3-4b27-a19d-315e172a872f","Type":"ContainerDied","Data":"0cb799aaa331ca1085e4b5f46df471214269591934f5272b2c34e29017351c8f"} Nov 03 17:44:31.006125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.006100 6001 scope.go:122] "RemoveContainer" containerID="4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3" Nov 03 17:44:31.006511 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:44:31.006488 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3\": container with ID starting with 4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3 not found: ID does not exist" containerID="4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3" Nov 03 17:44:31.006576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.006523 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3"} err="failed to get container status \"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3\": rpc error: code = NotFound desc = could not find container \"4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3\": container with ID starting with 4346863c33c28d9297a7b95abd24dfdc23eb535f3e6c966da2ec9023a5b129e3 not found: ID does not exist" Nov 03 17:44:31.006576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.006543 6001 scope.go:122] "RemoveContainer" containerID="78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f" Nov 03 17:44:31.015309 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.015285 6001 scope.go:122] "RemoveContainer" containerID="78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f" Nov 03 17:44:31.015628 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:44:31.015604 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f\": container with ID starting with 78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f not found: ID does not exist" containerID="78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f" Nov 03 17:44:31.015714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.015637 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f"} err="failed to get container status \"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f\": rpc error: code = NotFound desc = could not find container \"78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f\": container with ID starting with 78cd37e9861abb3e5fb3e7be927b4b917f89e98d09abf4c8ad15b4bf5df0fc9f not found: ID does not exist" Nov 03 17:44:31.052856 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.052758 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:31.065516 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.065479 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85bcq6r"] Nov 03 17:44:31.091070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.091032 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:31.097885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:31.097853 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7xvrsq"] Nov 03 17:44:32.788583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:32.788550 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e577a64-c6c3-4b27-a19d-315e172a872f" path="/var/lib/kubelet/pods/4e577a64-c6c3-4b27-a19d-315e172a872f/volumes" Nov 03 17:44:32.788991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:44:32.788945 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" path="/var/lib/kubelet/pods/e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54/volumes" Nov 03 17:44:52.251989 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:44:52.251945 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:45:00.161719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.161686 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162097 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4e577a64-c6c3-4b27-a19d-315e172a872f" containerName="payload-pre-processing" Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162107 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e577a64-c6c3-4b27-a19d-315e172a872f" containerName="payload-pre-processing" Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162124 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" containerName="revoke-keys" Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162129 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" containerName="revoke-keys" Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162146 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" containerName="payload-processing" Nov 03 17:45:00.162160 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162151 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" containerName="payload-processing" Nov 03 17:45:00.162476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162220 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e577a64-c6c3-4b27-a19d-315e172a872f" containerName="payload-pre-processing" Nov 03 17:45:00.162476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162231 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d63b08-f6a6-44ca-aa43-2514a1e07a9f" containerName="revoke-keys" Nov 03 17:45:00.162476 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.162239 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cc0ad0-31f7-4342-9ddc-51ebeb2eac54" containerName="payload-processing" Nov 03 17:45:00.165411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.165393 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.181081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.181061 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:45:00.194012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.193981 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:45:00.196136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.196111 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:00.269287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.269236 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f9vz\" (UniqueName: \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.269514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.269398 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.281737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.281711 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:45:00.370284 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.370235 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f9vz\" (UniqueName: \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.370521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.370350 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.371011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.370990 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.394477 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.394425 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f9vz\" (UniqueName: \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\") pod \"maas-api-key-cleanup-29895465-9r7kf\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.478549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.478520 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:00.784027 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:45:00.783988 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5af2c1a_0352_4201_8404_39897e5e4074.slice/crio-50016873b447c0411d39b3251f879d089f3974f8eaf2cdf98391c5b2a44482ac WatchSource:0}: Error finding container 50016873b447c0411d39b3251f879d089f3974f8eaf2cdf98391c5b2a44482ac: Status 404 returned error can't find the container with id 50016873b447c0411d39b3251f879d089f3974f8eaf2cdf98391c5b2a44482ac Nov 03 17:45:00.790370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:00.790345 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:01.164298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:01.164259 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerStarted","Data":"392c5562d794cedb292d1a539d4ac98e082122ad75c33904f03ec158dafe7d29"} Nov 03 17:45:01.164737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:01.164325 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerStarted","Data":"50016873b447c0411d39b3251f879d089f3974f8eaf2cdf98391c5b2a44482ac"} Nov 03 17:45:01.207137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:01.207074 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" podStartSLOduration=1.207057488 podStartE2EDuration="1.207057488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:01.206108996 +0000 UTC m=+5396.539322579" watchObservedRunningTime="2026-11-03 17:45:01.207057488 +0000 UTC m=+5396.540271060" Nov 03 17:45:02.171813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:02.171779 6001 generic.go:356] "Generic (PLEG): container finished" podID="c5af2c1a-0352-4201-8404-39897e5e4074" containerID="392c5562d794cedb292d1a539d4ac98e082122ad75c33904f03ec158dafe7d29" exitCode=60 Nov 03 17:45:02.172352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:02.171841 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerDied","Data":"392c5562d794cedb292d1a539d4ac98e082122ad75c33904f03ec158dafe7d29"} Nov 03 17:45:02.172352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:02.172107 6001 scope.go:122] "RemoveContainer" containerID="392c5562d794cedb292d1a539d4ac98e082122ad75c33904f03ec158dafe7d29" Nov 03 17:45:03.178950 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:03.178910 6001 generic.go:356] "Generic (PLEG): container finished" podID="c5af2c1a-0352-4201-8404-39897e5e4074" containerID="15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1" exitCode=60 Nov 03 17:45:03.179480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:03.178991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerDied","Data":"15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1"} Nov 03 17:45:03.179480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:03.179041 6001 scope.go:122] "RemoveContainer" containerID="392c5562d794cedb292d1a539d4ac98e082122ad75c33904f03ec158dafe7d29" Nov 03 17:45:03.179480 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:03.179322 6001 scope.go:122] "RemoveContainer" containerID="15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1" Nov 03 17:45:03.179694 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:45:03.179658 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-9r7kf_odh-ai-gateway-infra(c5af2c1a-0352-4201-8404-39897e5e4074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" Nov 03 17:45:04.187530 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:04.187500 6001 scope.go:122] "RemoveContainer" containerID="15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1" Nov 03 17:45:04.187957 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:45:04.187760 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-9r7kf_odh-ai-gateway-infra(c5af2c1a-0352-4201-8404-39897e5e4074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" Nov 03 17:45:05.895859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:05.895832 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:45:06.016338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:06.016310 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:45:06.029377 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:06.029341 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk"] Nov 03 17:45:06.041771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:06.041741 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-gp4pk"] Nov 03 17:45:06.788654 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:06.788619 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7910922f-c874-4180-a440-76c4428d75b0" path="/var/lib/kubelet/pods/7910922f-c874-4180-a440-76c4428d75b0/volumes" Nov 03 17:45:13.781405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:13.781371 6001 scope.go:122] "RemoveContainer" containerID="15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1" Nov 03 17:45:14.254367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:14.254335 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerStarted","Data":"7bd09af82ba8123d06b83eb1fb496478a28ba210f8939e9ec9110814072d0875"} Nov 03 17:45:14.605118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:14.605079 6001 scope.go:122] "RemoveContainer" containerID="99cc9caf893a51907bfa3ac8ca28c4621880ebf84689008f2f3986dbe8ea97a9" Nov 03 17:45:15.261131 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:15.261097 6001 generic.go:356] "Generic (PLEG): container finished" podID="c5af2c1a-0352-4201-8404-39897e5e4074" containerID="7bd09af82ba8123d06b83eb1fb496478a28ba210f8939e9ec9110814072d0875" exitCode=60 Nov 03 17:45:15.261596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:15.261169 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerDied","Data":"7bd09af82ba8123d06b83eb1fb496478a28ba210f8939e9ec9110814072d0875"} Nov 03 17:45:15.261596 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:15.261239 6001 scope.go:122] "RemoveContainer" containerID="15d0e331b55610d3a0ebf7662d5c7decaa109cc5954e3ca2a9633ef52f029dd1" Nov 03 17:45:15.261699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:15.261603 6001 scope.go:122] "RemoveContainer" containerID="7bd09af82ba8123d06b83eb1fb496478a28ba210f8939e9ec9110814072d0875" Nov 03 17:45:15.261882 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:45:15.261864 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-9r7kf_odh-ai-gateway-infra(c5af2c1a-0352-4201-8404-39897e5e4074)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" Nov 03 17:45:15.316145 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:15.316102 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:16.412021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.411990 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:16.480027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.479988 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\" (UniqueName: \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\") pod \"c5af2c1a-0352-4201-8404-39897e5e4074\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " Nov 03 17:45:16.480255 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.480040 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\") pod \"c5af2c1a-0352-4201-8404-39897e5e4074\" (UID: \"c5af2c1a-0352-4201-8404-39897e5e4074\") " Nov 03 17:45:16.480460 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.480411 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca" pod "c5af2c1a-0352-4201-8404-39897e5e4074" (UID: "c5af2c1a-0352-4201-8404-39897e5e4074"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:16.482301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.482272 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz" pod "c5af2c1a-0352-4201-8404-39897e5e4074" (UID: "c5af2c1a-0352-4201-8404-39897e5e4074"). InnerVolumeSpecName "kube-api-access-4f9vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:16.581097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.580998 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f9vz\" (UniqueName: \"kubernetes.io/projected/c5af2c1a-0352-4201-8404-39897e5e4074-kube-api-access-4f9vz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:16.581097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:16.581032 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5af2c1a-0352-4201-8404-39897e5e4074-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:17.274133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:17.274105 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" Nov 03 17:45:17.274301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:17.274140 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf" event={"ID":"c5af2c1a-0352-4201-8404-39897e5e4074","Type":"ContainerDied","Data":"50016873b447c0411d39b3251f879d089f3974f8eaf2cdf98391c5b2a44482ac"} Nov 03 17:45:17.274301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:17.274205 6001 scope.go:122] "RemoveContainer" containerID="7bd09af82ba8123d06b83eb1fb496478a28ba210f8939e9ec9110814072d0875" Nov 03 17:45:17.341755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:17.341714 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:17.347914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:17.347881 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-9r7kf"] Nov 03 17:45:18.788742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:18.788693 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" path="/var/lib/kubelet/pods/c5af2c1a-0352-4201-8404-39897e5e4074/volumes" Nov 03 17:45:22.254089 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:45:22.254046 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:45:26.782098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.782056 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8"] Nov 03 17:45:26.783176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783147 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.783176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783176 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.783344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783242 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.783344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783253 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.783416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783362 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.783416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.783382 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:26.787275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.787243 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.816787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.816741 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:45:26.817012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.816797 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:45:26.817012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.816874 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:45:26.817012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.816959 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:45:26.820017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.819983 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8"] Nov 03 17:45:26.888424 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.888364 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7jp\" (UniqueName: \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.888678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.888546 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.888678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.888647 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.989491 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.989428 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7jp\" (UniqueName: \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.989705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.989538 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.989705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.989581 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.990426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.990398 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:26.992177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:26.992147 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:27.014056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:27.014017 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7jp\" (UniqueName: \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\") pod \"maas-api-revoke-sub-3e31a506-8b6s8\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:27.105909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:27.105815 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:27.427778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:27.427731 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8"] Nov 03 17:45:27.429289 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:45:27.429260 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a5d742_40dd_497a_9e86_cc73ff94b32c.slice/crio-c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6 WatchSource:0}: Error finding container c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6: Status 404 returned error can't find the container with id c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6 Nov 03 17:45:28.418055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:28.418013 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" event={"ID":"a7a5d742-40dd-497a-9e86-cc73ff94b32c","Type":"ContainerStarted","Data":"46c09d3ee5dfcc5b9a49cdd9278e61625483c33daf75b1f472ae5ad5a5d14c8c"} Nov 03 17:45:28.418724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:28.418696 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" event={"ID":"a7a5d742-40dd-497a-9e86-cc73ff94b32c","Type":"ContainerStarted","Data":"c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6"} Nov 03 17:45:28.476656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:28.476587 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" podStartSLOduration=2.476564244 podStartE2EDuration="2.476564244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:28.474890608 +0000 UTC m=+5423.808104179" watchObservedRunningTime="2026-11-03 17:45:28.476564244 +0000 UTC m=+5423.809777814" Nov 03 17:45:29.427538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:29.427492 6001 generic.go:356] "Generic (PLEG): container finished" podID="a7a5d742-40dd-497a-9e86-cc73ff94b32c" containerID="46c09d3ee5dfcc5b9a49cdd9278e61625483c33daf75b1f472ae5ad5a5d14c8c" exitCode=0 Nov 03 17:45:29.427948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:29.427579 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" event={"ID":"a7a5d742-40dd-497a-9e86-cc73ff94b32c","Type":"ContainerDied","Data":"46c09d3ee5dfcc5b9a49cdd9278e61625483c33daf75b1f472ae5ad5a5d14c8c"} Nov 03 17:45:30.030235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.030195 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs"] Nov 03 17:45:30.032648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.032623 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-6qzrs"] Nov 03 17:45:30.578795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.578766 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:30.643010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.642977 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\" (UniqueName: \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\") pod \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " Nov 03 17:45:30.643207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.643053 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\") pod \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " Nov 03 17:45:30.643207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.643115 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\") pod \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\" (UID: \"a7a5d742-40dd-497a-9e86-cc73ff94b32c\") " Nov 03 17:45:30.643507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.643474 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca" pod "a7a5d742-40dd-497a-9e86-cc73ff94b32c" (UID: "a7a5d742-40dd-497a-9e86-cc73ff94b32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:30.643627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.643589 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:30.645234 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.645206 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca" pod "a7a5d742-40dd-497a-9e86-cc73ff94b32c" (UID: "a7a5d742-40dd-497a-9e86-cc73ff94b32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:30.645415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.645392 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp" pod "a7a5d742-40dd-497a-9e86-cc73ff94b32c" (UID: "a7a5d742-40dd-497a-9e86-cc73ff94b32c"). InnerVolumeSpecName "kube-api-access-2w7jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:30.744562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.744531 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7jp\" (UniqueName: \"kubernetes.io/projected/a7a5d742-40dd-497a-9e86-cc73ff94b32c-kube-api-access-2w7jp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:30.744562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.744559 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7a5d742-40dd-497a-9e86-cc73ff94b32c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:30.788383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:30.788350 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5260ea57-d51f-4b5c-90d3-58f52b68549b" path="/var/lib/kubelet/pods/5260ea57-d51f-4b5c-90d3-58f52b68549b/volumes" Nov 03 17:45:31.439350 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:31.439320 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" Nov 03 17:45:31.439564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:31.439359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8" event={"ID":"a7a5d742-40dd-497a-9e86-cc73ff94b32c","Type":"ContainerDied","Data":"c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6"} Nov 03 17:45:31.439564 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:31.439411 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c22a3b630c222eb8fd87c513ceeb632c7c3f62b6c3f182bbdd5ae6d1ecd69dc6" Nov 03 17:45:46.020988 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.020943 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5"] Nov 03 17:45:46.021473 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021456 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a7a5d742-40dd-497a-9e86-cc73ff94b32c" containerName="revoke-keys" Nov 03 17:45:46.021572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021474 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a5d742-40dd-497a-9e86-cc73ff94b32c" containerName="revoke-keys" Nov 03 17:45:46.021572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021515 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:46.021572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021524 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:46.021697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021640 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5af2c1a-0352-4201-8404-39897e5e4074" containerName="cleanup" Nov 03 17:45:46.021697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.021655 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a5d742-40dd-497a-9e86-cc73ff94b32c" containerName="revoke-keys" Nov 03 17:45:46.026322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.026298 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.044329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.044293 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:45:46.057101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.057046 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:45:46.057101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.057068 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:45:46.057383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.057117 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:45:46.059093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.059066 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5"] Nov 03 17:45:46.148861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.148819 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.149080 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.148985 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.149141 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.149075 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqs6m\" (UniqueName: \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.250529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.250484 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.250727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.250618 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqs6m\" (UniqueName: \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.250727 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.250712 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.251364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.251342 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.253087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.253066 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.274600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.274522 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqs6m\" (UniqueName: \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\") pod \"maas-api-revoke-sub-fe064636-lq7j5\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.339593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.339552 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:46.644597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:46.644565 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5"] Nov 03 17:45:46.646637 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:45:46.646605 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ccdbaf_0139_43dc_970d_cafc6b699911.slice/crio-5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93 WatchSource:0}: Error finding container 5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93: Status 404 returned error can't find the container with id 5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93 Nov 03 17:45:47.029930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:47.029890 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp"] Nov 03 17:45:47.042803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:47.042771 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v98sp"] Nov 03 17:45:47.560707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:47.560671 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" event={"ID":"c9ccdbaf-0139-43dc-970d-cafc6b699911","Type":"ContainerStarted","Data":"b1d926ce73e1d11c1b1f32ad68256447a8c7a4e529671511f0ed291dec2ae346"} Nov 03 17:45:47.560848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:47.560740 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" event={"ID":"c9ccdbaf-0139-43dc-970d-cafc6b699911","Type":"ContainerStarted","Data":"5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93"} Nov 03 17:45:48.567863 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.567828 6001 generic.go:356] "Generic (PLEG): container finished" podID="c9ccdbaf-0139-43dc-970d-cafc6b699911" containerID="b1d926ce73e1d11c1b1f32ad68256447a8c7a4e529671511f0ed291dec2ae346" exitCode=0 Nov 03 17:45:48.568308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.567885 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" event={"ID":"c9ccdbaf-0139-43dc-970d-cafc6b699911","Type":"ContainerDied","Data":"b1d926ce73e1d11c1b1f32ad68256447a8c7a4e529671511f0ed291dec2ae346"} Nov 03 17:45:48.710117 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.710086 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:48.776507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.776432 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\") pod \"c9ccdbaf-0139-43dc-970d-cafc6b699911\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " Nov 03 17:45:48.776741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.776565 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\") pod \"c9ccdbaf-0139-43dc-970d-cafc6b699911\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " Nov 03 17:45:48.776826 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.776771 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\" (UniqueName: \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\") pod \"c9ccdbaf-0139-43dc-970d-cafc6b699911\" (UID: \"c9ccdbaf-0139-43dc-970d-cafc6b699911\") " Nov 03 17:45:48.776946 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.776918 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca" pod "c9ccdbaf-0139-43dc-970d-cafc6b699911" (UID: "c9ccdbaf-0139-43dc-970d-cafc6b699911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:48.777077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.777061 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:48.778734 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.778703 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca" pod "c9ccdbaf-0139-43dc-970d-cafc6b699911" (UID: "c9ccdbaf-0139-43dc-970d-cafc6b699911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:48.778885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.778863 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m" pod "c9ccdbaf-0139-43dc-970d-cafc6b699911" (UID: "c9ccdbaf-0139-43dc-970d-cafc6b699911"). InnerVolumeSpecName "kube-api-access-pqs6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:48.790802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.790763 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5890fb3-d052-458a-ad8e-dab457674fc5" path="/var/lib/kubelet/pods/a5890fb3-d052-458a-ad8e-dab457674fc5/volumes" Nov 03 17:45:48.878058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.877966 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqs6m\" (UniqueName: \"kubernetes.io/projected/c9ccdbaf-0139-43dc-970d-cafc6b699911-kube-api-access-pqs6m\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:48.878058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:48.878000 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ccdbaf-0139-43dc-970d-cafc6b699911-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:45:49.574895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:49.574865 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" Nov 03 17:45:49.574895 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:49.574875 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5" event={"ID":"c9ccdbaf-0139-43dc-970d-cafc6b699911","Type":"ContainerDied","Data":"5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93"} Nov 03 17:45:49.575391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:49.574919 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5eaba4ff88f5ec7a9568cc9d4cf0f3bdd134403349cfc25a7f57c108143b6c93" Nov 03 17:45:52.250913 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:45:52.250871 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:45:57.029858 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:57.029820 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv"] Nov 03 17:45:57.042125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:57.042089 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nvspv"] Nov 03 17:45:58.789205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:45:58.789168 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ebb6c6-c821-4c18-aa7f-f03f8ed13595" path="/var/lib/kubelet/pods/e7ebb6c6-c821-4c18-aa7f-f03f8ed13595/volumes" Nov 03 17:46:01.801818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.801781 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:01.802256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.802238 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c9ccdbaf-0139-43dc-970d-cafc6b699911" containerName="revoke-keys" Nov 03 17:46:01.802256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.802250 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ccdbaf-0139-43dc-970d-cafc6b699911" containerName="revoke-keys" Nov 03 17:46:01.802340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.802325 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ccdbaf-0139-43dc-970d-cafc6b699911" containerName="revoke-keys" Nov 03 17:46:01.805849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.805831 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:01.826010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.825972 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:01.919435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.919391 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:01.919656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:01.919543 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqvpb\" (UniqueName: \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.021211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.021167 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqvpb\" (UniqueName: \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.021414 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.021294 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.022021 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.022002 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.045944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.045904 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqvpb\" (UniqueName: \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\") pod \"payload-pre-processing-7995766b66-djzk5\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.118660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.118568 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.420693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.420659 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:02.421647 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:46:02.421620 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726de118_40af_4623_9d29_a59eba0447cf.slice/crio-93488edb1f163e468050ca35c6cf725a13f01fdf5f734ed0a27a3be2909315e1 WatchSource:0}: Error finding container 93488edb1f163e468050ca35c6cf725a13f01fdf5f734ed0a27a3be2909315e1: Status 404 returned error can't find the container with id 93488edb1f163e468050ca35c6cf725a13f01fdf5f734ed0a27a3be2909315e1 Nov 03 17:46:02.645587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.645545 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" event={"ID":"726de118-40af-4623-9d29-a59eba0447cf","Type":"ContainerStarted","Data":"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9"} Nov 03 17:46:02.645818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.645614 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" event={"ID":"726de118-40af-4623-9d29-a59eba0447cf","Type":"ContainerStarted","Data":"93488edb1f163e468050ca35c6cf725a13f01fdf5f734ed0a27a3be2909315e1"} Nov 03 17:46:02.645818 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.645663 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:02.689238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:02.689128 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" podStartSLOduration=1.689112992 podStartE2EDuration="1.689112992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:02.687683643 +0000 UTC m=+5458.020897217" watchObservedRunningTime="2026-11-03 17:46:02.689112992 +0000 UTC m=+5458.022326558" Nov 03 17:46:13.655689 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:13.655658 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:13.727385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:13.727348 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 17:46:14.652012 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:14.651974 6001 scope.go:122] "RemoveContainer" containerID="48bcf0a05e7749a33af592ae8ca959804274c1950d05699031e3cd28722c4ec7" Nov 03 17:46:14.661204 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:14.661181 6001 scope.go:122] "RemoveContainer" containerID="e4b98f130b1dbc9428bf3d7e1601a5c917bb14619630768f8745192cdd2c5584" Nov 03 17:46:15.397526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.397492 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:15.403051 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.403027 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.423882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.423851 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:15.449367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.449329 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5x8\" (UniqueName: \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.449619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.449511 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.550673 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.550626 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5x8\" (UniqueName: \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.550877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.550758 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.551524 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.551501 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.576426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.576392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5x8\" (UniqueName: \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\") pod \"payload-processing-58865ff9d-xtb24\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:15.716808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:15.716719 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:16.019114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:16.019085 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:16.020599 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:46:16.020563 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58f24395_58ff_44f8_9785_c47429d1a4cf.slice/crio-c9c2010cc0d8bcd641ffb4a079beccde1efc1d4fee6142febe14dfe31210eb7a WatchSource:0}: Error finding container c9c2010cc0d8bcd641ffb4a079beccde1efc1d4fee6142febe14dfe31210eb7a: Status 404 returned error can't find the container with id c9c2010cc0d8bcd641ffb4a079beccde1efc1d4fee6142febe14dfe31210eb7a Nov 03 17:46:16.730266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:16.730228 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" event={"ID":"58f24395-58ff-44f8-9785-c47429d1a4cf","Type":"ContainerStarted","Data":"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056"} Nov 03 17:46:16.730738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:16.730300 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:16.730738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:16.730324 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" event={"ID":"58f24395-58ff-44f8-9785-c47429d1a4cf","Type":"ContainerStarted","Data":"c9c2010cc0d8bcd641ffb4a079beccde1efc1d4fee6142febe14dfe31210eb7a"} Nov 03 17:46:16.773868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:16.773810 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" podStartSLOduration=1.7737933190000001 podStartE2EDuration="1.773793319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:16.772380249 +0000 UTC m=+5472.105593817" watchObservedRunningTime="2026-11-03 17:46:16.773793319 +0000 UTC m=+5472.107006912" Nov 03 17:46:18.728515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:18.728464 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" podUID="f4ceab1b-3778-4b26-a436-d59dbbe65177" containerName="payload-pre-processing" containerID="cri-o://2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50" gracePeriod=25 Nov 03 17:46:18.987209 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:18.987144 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 17:46:19.089044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.089004 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\") pod \"f4ceab1b-3778-4b26-a436-d59dbbe65177\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " Nov 03 17:46:19.089272 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.089105 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\" (UniqueName: \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\") pod \"f4ceab1b-3778-4b26-a436-d59dbbe65177\" (UID: \"f4ceab1b-3778-4b26-a436-d59dbbe65177\") " Nov 03 17:46:19.089405 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.089382 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume" pod "f4ceab1b-3778-4b26-a436-d59dbbe65177" (UID: "f4ceab1b-3778-4b26-a436-d59dbbe65177"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:19.089591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.089575 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ceab1b-3778-4b26-a436-d59dbbe65177-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:19.091335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.091311 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k" pod "f4ceab1b-3778-4b26-a436-d59dbbe65177" (UID: "f4ceab1b-3778-4b26-a436-d59dbbe65177"). InnerVolumeSpecName "kube-api-access-vgj6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:19.191022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.190983 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgj6k\" (UniqueName: \"kubernetes.io/projected/f4ceab1b-3778-4b26-a436-d59dbbe65177-kube-api-access-vgj6k\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:19.748039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.748000 6001 generic.go:356] "Generic (PLEG): container finished" podID="f4ceab1b-3778-4b26-a436-d59dbbe65177" containerID="2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50" exitCode=0 Nov 03 17:46:19.748504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.748044 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" Nov 03 17:46:19.748504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.748047 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" event={"ID":"f4ceab1b-3778-4b26-a436-d59dbbe65177","Type":"ContainerDied","Data":"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50"} Nov 03 17:46:19.748504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.748169 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-gblz5" event={"ID":"f4ceab1b-3778-4b26-a436-d59dbbe65177","Type":"ContainerDied","Data":"0fc2fcd8652df32fee17124bc81defedebf05a4d35d76b6b27a2bc9d064780f0"} Nov 03 17:46:19.748504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.748196 6001 scope.go:122] "RemoveContainer" containerID="2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50" Nov 03 17:46:19.758308 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.758289 6001 scope.go:122] "RemoveContainer" containerID="2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50" Nov 03 17:46:19.758609 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:19.758581 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50\": container with ID starting with 2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50 not found: ID does not exist" containerID="2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50" Nov 03 17:46:19.758691 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.758621 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50"} err="failed to get container status \"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50\": rpc error: code = NotFound desc = could not find container \"2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50\": container with ID starting with 2459aca5b086c7502c1dfd8604d2bec2b0369e2f9ab9663df675df5b5c4cea50 not found: ID does not exist" Nov 03 17:46:19.800987 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.800943 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 17:46:19.813311 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:19.813274 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-gblz5"] Nov 03 17:46:20.786995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:20.786960 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ceab1b-3778-4b26-a436-d59dbbe65177" path="/var/lib/kubelet/pods/f4ceab1b-3778-4b26-a436-d59dbbe65177/volumes" Nov 03 17:46:22.251796 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:22.251757 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:46:27.739591 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:27.739545 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:27.811226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:27.811181 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 17:46:29.703077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.703041 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6d85d667ff-kc82s"] Nov 03 17:46:29.703588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.703474 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f4ceab1b-3778-4b26-a436-d59dbbe65177" containerName="payload-pre-processing" Nov 03 17:46:29.703588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.703487 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ceab1b-3778-4b26-a436-d59dbbe65177" containerName="payload-pre-processing" Nov 03 17:46:29.703682 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.703600 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ceab1b-3778-4b26-a436-d59dbbe65177" containerName="payload-pre-processing" Nov 03 17:46:29.706656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.706637 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.728246 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.728223 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6d85d667ff-kc82s"] Nov 03 17:46:29.804556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.804523 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139f8c8-371a-44d5-8d0e-636f0eb11e48-plugins-config-volume\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.804783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.804569 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv2zq\" (UniqueName: \"kubernetes.io/projected/9139f8c8-371a-44d5-8d0e-636f0eb11e48-kube-api-access-kv2zq\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.905583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.905551 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139f8c8-371a-44d5-8d0e-636f0eb11e48-plugins-config-volume\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.905773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.905609 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv2zq\" (UniqueName: \"kubernetes.io/projected/9139f8c8-371a-44d5-8d0e-636f0eb11e48-kube-api-access-kv2zq\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.906560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.906538 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9139f8c8-371a-44d5-8d0e-636f0eb11e48-plugins-config-volume\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:29.929675 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:29.929641 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv2zq\" (UniqueName: \"kubernetes.io/projected/9139f8c8-371a-44d5-8d0e-636f0eb11e48-kube-api-access-kv2zq\") pod \"payload-processing-6d85d667ff-kc82s\" (UID: \"9139f8c8-371a-44d5-8d0e-636f0eb11e48\") " pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:30.027835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.027800 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:30.330273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.330240 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6d85d667ff-kc82s"] Nov 03 17:46:30.331032 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:46:30.331005 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9139f8c8_371a_44d5_8d0e_636f0eb11e48.slice/crio-0a9f373a5351d1cc8e35e812e6b0ae19d7a3e075fba5e85412efdb737f0d6d13 WatchSource:0}: Error finding container 0a9f373a5351d1cc8e35e812e6b0ae19d7a3e075fba5e85412efdb737f0d6d13: Status 404 returned error can't find the container with id 0a9f373a5351d1cc8e35e812e6b0ae19d7a3e075fba5e85412efdb737f0d6d13 Nov 03 17:46:30.810086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.810044 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" event={"ID":"9139f8c8-371a-44d5-8d0e-636f0eb11e48","Type":"ContainerStarted","Data":"b551bfffc2a4c049c617560e557b1d7bccdeda1219ac3e83eba70943a2b6eb6e"} Nov 03 17:46:30.810537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.810110 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" event={"ID":"9139f8c8-371a-44d5-8d0e-636f0eb11e48","Type":"ContainerStarted","Data":"0a9f373a5351d1cc8e35e812e6b0ae19d7a3e075fba5e85412efdb737f0d6d13"} Nov 03 17:46:30.810537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.810151 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:30.854910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:30.854858 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" podStartSLOduration=1.854842631 podStartE2EDuration="1.854842631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:30.853455605 +0000 UTC m=+5486.186669166" watchObservedRunningTime="2026-11-03 17:46:30.854842631 +0000 UTC m=+5486.188056203" Nov 03 17:46:32.811609 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:32.811532 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" podUID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" containerName="payload-processing" containerID="cri-o://2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228" gracePeriod=25 Nov 03 17:46:33.081886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.081857 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 17:46:33.239194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.239161 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\" (UniqueName: \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\") pod \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " Nov 03 17:46:33.239386 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.239212 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\") pod \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\" (UID: \"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9\") " Nov 03 17:46:33.239585 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.239565 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume" pod "e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" (UID: "e1e3af6e-bac6-4c92-b299-f8c669b9a8d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:33.241307 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.241287 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5" pod "e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" (UID: "e1e3af6e-bac6-4c92-b299-f8c669b9a8d9"). InnerVolumeSpecName "kube-api-access-8h4n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:33.340909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.340813 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4n5\" (UniqueName: \"kubernetes.io/projected/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-kube-api-access-8h4n5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:33.340909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.340850 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:33.827714 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.827681 6001 generic.go:356] "Generic (PLEG): container finished" podID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" containerID="2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228" exitCode=0 Nov 03 17:46:33.828159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.827731 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" Nov 03 17:46:33.828159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.827769 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" event={"ID":"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9","Type":"ContainerDied","Data":"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228"} Nov 03 17:46:33.828159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.827829 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-jkphl" event={"ID":"e1e3af6e-bac6-4c92-b299-f8c669b9a8d9","Type":"ContainerDied","Data":"50c97516da6c3f9b92bafba617eae9c1b7d93613eaf62d229549a4f710b3d6ea"} Nov 03 17:46:33.828159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.827853 6001 scope.go:122] "RemoveContainer" containerID="2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228" Nov 03 17:46:33.838165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.838147 6001 scope.go:122] "RemoveContainer" containerID="2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228" Nov 03 17:46:33.838506 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:33.838487 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228\": container with ID starting with 2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228 not found: ID does not exist" containerID="2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228" Nov 03 17:46:33.838565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.838513 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228"} err="failed to get container status \"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228\": rpc error: code = NotFound desc = could not find container \"2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228\": container with ID starting with 2890f154e7c73f405b55b48ddb0ffae553f53f34afe4634c96b670f0fed7b228 not found: ID does not exist" Nov 03 17:46:33.880982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.880941 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 17:46:33.893547 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:33.893510 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-jkphl"] Nov 03 17:46:34.789241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:34.789206 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" path="/var/lib/kubelet/pods/e1e3af6e-bac6-4c92-b299-f8c669b9a8d9/volumes" Nov 03 17:46:41.818526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:41.818495 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6d85d667ff-kc82s" Nov 03 17:46:41.897372 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:41.897339 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:41.946759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:41.946724 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:45.341401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.341363 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:46:45.341862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.341838 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" containerName="payload-processing" Nov 03 17:46:45.341862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.341852 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" containerName="payload-processing" Nov 03 17:46:45.341954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.341934 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e3af6e-bac6-4c92-b299-f8c669b9a8d9" containerName="payload-processing" Nov 03 17:46:45.346646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.346627 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.362373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.362354 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 17:46:45.365162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.365136 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-6krb6\"" Nov 03 17:46:45.375227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.375206 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 17:46:45.377733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.377711 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 17:46:45.377850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.377781 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 17:46:45.377850 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.377782 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 17:46:45.379521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.379497 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:46:45.464973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.464928 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.465208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.465029 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.465208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.465143 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94f4w\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.465326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.465245 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.565994 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.565957 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.566179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.566009 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94f4w\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.566179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.566129 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.566304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.566233 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.566920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.566898 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.567041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.566903 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.568638 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.568617 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.590254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.590213 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94f4w\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\") pod \"payload-processing-external-model-5bcd7997-sg5lb\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.660570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.660487 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:45.965436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:45.965407 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:46:45.965963 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:46:45.965933 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd69e54c_2ac3_4562_83d7_557e5b38ff2c.slice/crio-b1261f8e1703fe22f6c523bb2e2124f2e81fe233e009e2b9916bc3381c9634ed WatchSource:0}: Error finding container b1261f8e1703fe22f6c523bb2e2124f2e81fe233e009e2b9916bc3381c9634ed: Status 404 returned error can't find the container with id b1261f8e1703fe22f6c523bb2e2124f2e81fe233e009e2b9916bc3381c9634ed Nov 03 17:46:46.898529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.898488 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" podUID="58f24395-58ff-44f8-9785-c47429d1a4cf" containerName="payload-processing" containerID="cri-o://91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056" gracePeriod=25 Nov 03 17:46:46.902962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.902921 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" event={"ID":"cd69e54c-2ac3-4562-83d7-557e5b38ff2c","Type":"ContainerStarted","Data":"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0"} Nov 03 17:46:46.903130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.902992 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" event={"ID":"cd69e54c-2ac3-4562-83d7-557e5b38ff2c","Type":"ContainerStarted","Data":"b1261f8e1703fe22f6c523bb2e2124f2e81fe233e009e2b9916bc3381c9634ed"} Nov 03 17:46:46.903192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.903128 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:46:46.947458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.947401 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" podUID="726de118-40af-4623-9d29-a59eba0447cf" containerName="payload-pre-processing" containerID="cri-o://d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9" gracePeriod=25 Nov 03 17:46:46.947610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:46.947484 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" podStartSLOduration=1.947471556 podStartE2EDuration="1.947471556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:46.947024935 +0000 UTC m=+5502.280238506" watchObservedRunningTime="2026-11-03 17:46:46.947471556 +0000 UTC m=+5502.280685116" Nov 03 17:46:47.260724 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.260699 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:47.274237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.274184 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:47.286559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.286530 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\" (UniqueName: \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\") pod \"726de118-40af-4623-9d29-a59eba0447cf\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " Nov 03 17:46:47.286705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.286625 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\" (UniqueName: \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\") pod \"58f24395-58ff-44f8-9785-c47429d1a4cf\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " Nov 03 17:46:47.286705 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.286671 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\") pod \"58f24395-58ff-44f8-9785-c47429d1a4cf\" (UID: \"58f24395-58ff-44f8-9785-c47429d1a4cf\") " Nov 03 17:46:47.286849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.286761 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\") pod \"726de118-40af-4623-9d29-a59eba0447cf\" (UID: \"726de118-40af-4623-9d29-a59eba0447cf\") " Nov 03 17:46:47.287294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.287174 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume" pod "58f24395-58ff-44f8-9785-c47429d1a4cf" (UID: "58f24395-58ff-44f8-9785-c47429d1a4cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:47.287294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.287258 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume" pod "726de118-40af-4623-9d29-a59eba0447cf" (UID: "726de118-40af-4623-9d29-a59eba0447cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:46:47.289078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.289051 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8" pod "58f24395-58ff-44f8-9785-c47429d1a4cf" (UID: "58f24395-58ff-44f8-9785-c47429d1a4cf"). InnerVolumeSpecName "kube-api-access-wd5x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:47.289166 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.289091 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb" pod "726de118-40af-4623-9d29-a59eba0447cf" (UID: "726de118-40af-4623-9d29-a59eba0447cf"). InnerVolumeSpecName "kube-api-access-wqvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:46:47.388237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.388196 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd5x8\" (UniqueName: \"kubernetes.io/projected/58f24395-58ff-44f8-9785-c47429d1a4cf-kube-api-access-wd5x8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:47.388237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.388240 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58f24395-58ff-44f8-9785-c47429d1a4cf-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:47.388550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.388258 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/726de118-40af-4623-9d29-a59eba0447cf-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:47.388550 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.388272 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqvpb\" (UniqueName: \"kubernetes.io/projected/726de118-40af-4623-9d29-a59eba0447cf-kube-api-access-wqvpb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:46:47.913074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.912975 6001 generic.go:356] "Generic (PLEG): container finished" podID="726de118-40af-4623-9d29-a59eba0447cf" containerID="d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9" exitCode=0 Nov 03 17:46:47.913074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.913044 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" Nov 03 17:46:47.913074 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.913050 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" event={"ID":"726de118-40af-4623-9d29-a59eba0447cf","Type":"ContainerDied","Data":"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9"} Nov 03 17:46:47.913666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.913112 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7995766b66-djzk5" event={"ID":"726de118-40af-4623-9d29-a59eba0447cf","Type":"ContainerDied","Data":"93488edb1f163e468050ca35c6cf725a13f01fdf5f734ed0a27a3be2909315e1"} Nov 03 17:46:47.913666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.913141 6001 scope.go:122] "RemoveContainer" containerID="d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9" Nov 03 17:46:47.915374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.915350 6001 generic.go:356] "Generic (PLEG): container finished" podID="58f24395-58ff-44f8-9785-c47429d1a4cf" containerID="91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056" exitCode=0 Nov 03 17:46:47.915534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.915508 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" Nov 03 17:46:47.915615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.915389 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" event={"ID":"58f24395-58ff-44f8-9785-c47429d1a4cf","Type":"ContainerDied","Data":"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056"} Nov 03 17:46:47.915615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.915604 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58865ff9d-xtb24" event={"ID":"58f24395-58ff-44f8-9785-c47429d1a4cf","Type":"ContainerDied","Data":"c9c2010cc0d8bcd641ffb4a079beccde1efc1d4fee6142febe14dfe31210eb7a"} Nov 03 17:46:47.925206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.925183 6001 scope.go:122] "RemoveContainer" containerID="d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9" Nov 03 17:46:47.926210 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:47.926177 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9\": container with ID starting with d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9 not found: ID does not exist" containerID="d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9" Nov 03 17:46:47.926297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.926223 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9"} err="failed to get container status \"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9\": rpc error: code = NotFound desc = could not find container \"d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9\": container with ID starting with d96814b5cc3cc3d65ddceb0ffb1f18f918ffcc7e7b7c56bc74be665919e662f9 not found: ID does not exist" Nov 03 17:46:47.926297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.926247 6001 scope.go:122] "RemoveContainer" containerID="91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056" Nov 03 17:46:47.937119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.937092 6001 scope.go:122] "RemoveContainer" containerID="91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056" Nov 03 17:46:47.937505 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:47.937475 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056\": container with ID starting with 91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056 not found: ID does not exist" containerID="91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056" Nov 03 17:46:47.937639 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.937515 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056"} err="failed to get container status \"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056\": rpc error: code = NotFound desc = could not find container \"91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056\": container with ID starting with 91a25f4e9b657b40833d1dd66459c393b89c8e35fb68e5ed996e774bcb8ad056 not found: ID does not exist" Nov 03 17:46:47.966678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.966643 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:47.980312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:47.980276 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7995766b66-djzk5"] Nov 03 17:46:48.025848 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:48.025779 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:48.029023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:48.028995 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-58865ff9d-xtb24"] Nov 03 17:46:48.801469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:48.800493 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58f24395-58ff-44f8-9785-c47429d1a4cf" path="/var/lib/kubelet/pods/58f24395-58ff-44f8-9785-c47429d1a4cf/volumes" Nov 03 17:46:48.810488 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:48.808494 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726de118-40af-4623-9d29-a59eba0447cf" path="/var/lib/kubelet/pods/726de118-40af-4623-9d29-a59eba0447cf/volumes" Nov 03 17:46:52.250532 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:46:52.250484 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:46:57.919052 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:46:57.919022 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:47:14.696570 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:47:14.696532 6001 scope.go:122] "RemoveContainer" containerID="3c06c9a1f1e5960851200c0a23523659ebecb5a9a9391fad66123c43e9d62bb0" Nov 03 17:47:14.705552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:47:14.705530 6001 scope.go:122] "RemoveContainer" containerID="43a682d238a3092829d429d6778fd2af729851ee756b6ec3d15653473176cec8" Nov 03 17:47:14.717214 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:47:14.717181 6001 scope.go:122] "RemoveContainer" containerID="f5dd9f067b701039f7f19c286e27601507ce0d45c494c6f710ab062cbe5123cc" Nov 03 17:47:22.249672 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:47:22.249635 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:47:52.249732 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:47:52.249686 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:48:14.062795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.062757 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj"] Nov 03 17:48:14.063299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063190 6001 cpu_manager.go:414] "containerMap: removing container" podUID="58f24395-58ff-44f8-9785-c47429d1a4cf" containerName="payload-processing" Nov 03 17:48:14.063299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063202 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58f24395-58ff-44f8-9785-c47429d1a4cf" containerName="payload-processing" Nov 03 17:48:14.063299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063217 6001 cpu_manager.go:414] "containerMap: removing container" podUID="726de118-40af-4623-9d29-a59eba0447cf" containerName="payload-pre-processing" Nov 03 17:48:14.063299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063222 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726de118-40af-4623-9d29-a59eba0447cf" containerName="payload-pre-processing" Nov 03 17:48:14.063299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063298 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="58f24395-58ff-44f8-9785-c47429d1a4cf" containerName="payload-processing" Nov 03 17:48:14.063504 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.063307 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="726de118-40af-4623-9d29-a59eba0447cf" containerName="payload-pre-processing" Nov 03 17:48:14.065954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.065926 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.080993 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.080970 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:48:14.093565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.093532 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:48:14.093565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.093532 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:48:14.093790 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.093617 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:48:14.095554 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.095525 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj"] Nov 03 17:48:14.121655 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.121593 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhwv\" (UniqueName: \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.121896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.121729 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.121896 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.121866 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.223078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.223038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.223275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.223119 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhwv\" (UniqueName: \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.223275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.223151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.223788 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.223765 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.225506 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.225476 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.246947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.246917 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhwv\" (UniqueName: \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\") pod \"maas-api-revoke-sub-e2547622-5m6vj\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.378811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.378720 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:14.683679 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:14.683651 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj"] Nov 03 17:48:14.684009 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:48:14.683983 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda932eec1_e0be_406b_bd63_38e45a935373.slice/crio-7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794 WatchSource:0}: Error finding container 7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794: Status 404 returned error can't find the container with id 7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794 Nov 03 17:48:15.405274 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:15.405230 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" event={"ID":"a932eec1-e0be-406b-bd63-38e45a935373","Type":"ContainerStarted","Data":"58c51bb08131414437a90ab7c0a95d7ea1ca44e8cde1e86ae128a526f49ace12"} Nov 03 17:48:15.405726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:15.405288 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" event={"ID":"a932eec1-e0be-406b-bd63-38e45a935373","Type":"ContainerStarted","Data":"7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794"} Nov 03 17:48:15.449751 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:15.449693 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" podStartSLOduration=1.449680135 podStartE2EDuration="1.449680135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:15.448502239 +0000 UTC m=+5590.781715808" watchObservedRunningTime="2026-11-03 17:48:15.449680135 +0000 UTC m=+5590.782893697" Nov 03 17:48:16.412197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:16.412110 6001 generic.go:356] "Generic (PLEG): container finished" podID="a932eec1-e0be-406b-bd63-38e45a935373" containerID="58c51bb08131414437a90ab7c0a95d7ea1ca44e8cde1e86ae128a526f49ace12" exitCode=0 Nov 03 17:48:16.412649 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:16.412198 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" event={"ID":"a932eec1-e0be-406b-bd63-38e45a935373","Type":"ContainerDied","Data":"58c51bb08131414437a90ab7c0a95d7ea1ca44e8cde1e86ae128a526f49ace12"} Nov 03 17:48:17.569155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.569121 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:17.658031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.657995 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\") pod \"a932eec1-e0be-406b-bd63-38e45a935373\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " Nov 03 17:48:17.658227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.658080 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\" (UniqueName: \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\") pod \"a932eec1-e0be-406b-bd63-38e45a935373\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " Nov 03 17:48:17.658227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.658169 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\") pod \"a932eec1-e0be-406b-bd63-38e45a935373\" (UID: \"a932eec1-e0be-406b-bd63-38e45a935373\") " Nov 03 17:48:17.658545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.658515 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca" pod "a932eec1-e0be-406b-bd63-38e45a935373" (UID: "a932eec1-e0be-406b-bd63-38e45a935373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:17.660326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.660294 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca" pod "a932eec1-e0be-406b-bd63-38e45a935373" (UID: "a932eec1-e0be-406b-bd63-38e45a935373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:48:17.660438 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.660299 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv" pod "a932eec1-e0be-406b-bd63-38e45a935373" (UID: "a932eec1-e0be-406b-bd63-38e45a935373"). InnerVolumeSpecName "kube-api-access-jvhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:17.759433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.759393 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:17.759433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.759426 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a932eec1-e0be-406b-bd63-38e45a935373-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:17.759433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:17.759439 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvhwv\" (UniqueName: \"kubernetes.io/projected/a932eec1-e0be-406b-bd63-38e45a935373-kube-api-access-jvhwv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:18.424559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:18.424531 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" Nov 03 17:48:18.424750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:18.424529 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj" event={"ID":"a932eec1-e0be-406b-bd63-38e45a935373","Type":"ContainerDied","Data":"7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794"} Nov 03 17:48:18.424750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:18.424667 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dccfbcee575a67a9bf695056eccab70e0c513b5d7ae383923888d9501936794" Nov 03 17:48:19.028902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:19.028863 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2"] Nov 03 17:48:19.041419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:19.041386 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b73efcce-9nmc2"] Nov 03 17:48:20.788420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:20.788382 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d96a8f-873e-4e99-8a0c-80eaf4b6234f" path="/var/lib/kubelet/pods/a3d96a8f-873e-4e99-8a0c-80eaf4b6234f/volumes" Nov 03 17:48:22.251668 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:48:22.251632 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:48:24.826103 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:24.826067 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:48:29.827363 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:29.827306 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" podUID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" containerName="payload-processing" containerID="cri-o://64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0" gracePeriod=25 Nov 03 17:48:30.089240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.089173 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:48:30.222907 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.222875 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\") pod \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " Nov 03 17:48:30.223125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.222964 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\") pod \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " Nov 03 17:48:30.223125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.223068 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\") pod \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " Nov 03 17:48:30.223240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.223222 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\") pod \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\" (UID: \"cd69e54c-2ac3-4562-83d7-557e5b38ff2c\") " Nov 03 17:48:30.223302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.223281 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume" pod "cd69e54c-2ac3-4562-83d7-557e5b38ff2c" (UID: "cd69e54c-2ac3-4562-83d7-557e5b38ff2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:30.223822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.223660 6001 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-routing-overlay-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:30.223822 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.223784 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume" pod "cd69e54c-2ac3-4562-83d7-557e5b38ff2c" (UID: "cd69e54c-2ac3-4562-83d7-557e5b38ff2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:30.225298 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.225276 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials" pod "cd69e54c-2ac3-4562-83d7-557e5b38ff2c" (UID: "cd69e54c-2ac3-4562-83d7-557e5b38ff2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:30.225431 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.225346 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w" pod "cd69e54c-2ac3-4562-83d7-557e5b38ff2c" (UID: "cd69e54c-2ac3-4562-83d7-557e5b38ff2c"). InnerVolumeSpecName "kube-api-access-94f4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:30.324890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.324840 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94f4w\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-kube-api-access-94f4w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:30.324890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.324878 6001 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-provider-credentials\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:30.324890 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.324888 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd69e54c-2ac3-4562-83d7-557e5b38ff2c-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:30.506823 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.506782 6001 generic.go:356] "Generic (PLEG): container finished" podID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" containerID="64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0" exitCode=0 Nov 03 17:48:30.507050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.506843 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" Nov 03 17:48:30.507050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.506846 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" event={"ID":"cd69e54c-2ac3-4562-83d7-557e5b38ff2c","Type":"ContainerDied","Data":"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0"} Nov 03 17:48:30.507050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.506973 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb" event={"ID":"cd69e54c-2ac3-4562-83d7-557e5b38ff2c","Type":"ContainerDied","Data":"b1261f8e1703fe22f6c523bb2e2124f2e81fe233e009e2b9916bc3381c9634ed"} Nov 03 17:48:30.507050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.506996 6001 scope.go:122] "RemoveContainer" containerID="64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0" Nov 03 17:48:30.516831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.516809 6001 scope.go:122] "RemoveContainer" containerID="64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0" Nov 03 17:48:30.517195 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:48:30.517168 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0\": container with ID starting with 64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0 not found: ID does not exist" containerID="64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0" Nov 03 17:48:30.517280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.517207 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0"} err="failed to get container status \"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0\": rpc error: code = NotFound desc = could not find container \"64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0\": container with ID starting with 64ad7a9afe90fe64fa07317ad167b0099ccd6504ad3be1874e51d251b2ab8bb0 not found: ID does not exist" Nov 03 17:48:30.560087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.560052 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:48:30.564538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.564506 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-sg5lb"] Nov 03 17:48:30.788412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:30.788320 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" path="/var/lib/kubelet/pods/cd69e54c-2ac3-4562-83d7-557e5b38ff2c/volumes" Nov 03 17:48:37.955708 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.955673 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6"] Nov 03 17:48:37.956124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956083 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a932eec1-e0be-406b-bd63-38e45a935373" containerName="revoke-keys" Nov 03 17:48:37.956124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956094 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a932eec1-e0be-406b-bd63-38e45a935373" containerName="revoke-keys" Nov 03 17:48:37.956124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956112 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" containerName="payload-processing" Nov 03 17:48:37.956124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956118 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" containerName="payload-processing" Nov 03 17:48:37.956266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956193 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a932eec1-e0be-406b-bd63-38e45a935373" containerName="revoke-keys" Nov 03 17:48:37.956266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.956203 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd69e54c-2ac3-4562-83d7-557e5b38ff2c" containerName="payload-processing" Nov 03 17:48:37.960879 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.960860 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:37.992693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.992665 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:48:37.992693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.992666 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:48:37.992693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.992670 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:48:37.992920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.992758 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:48:37.994469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:37.994435 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6"] Nov 03 17:48:38.038998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.038968 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68qr\" (UniqueName: \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.039161 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.039025 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.039213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.039154 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.139911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.139879 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.140067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.139970 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.140067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.140010 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68qr\" (UniqueName: \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.140695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.140672 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.142205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.142183 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.163857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.163827 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68qr\" (UniqueName: \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\") pod \"maas-api-revoke-sub-c1e271a9-q4gw6\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.274365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.274321 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:38.575514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:38.575481 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6"] Nov 03 17:48:38.577004 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:48:38.576977 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3a650b_034a_494a_84f2_d91c5b6acb86.slice/crio-4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37 WatchSource:0}: Error finding container 4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37: Status 404 returned error can't find the container with id 4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37 Nov 03 17:48:39.570302 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:39.570267 6001 generic.go:356] "Generic (PLEG): container finished" podID="2c3a650b-034a-494a-84f2-d91c5b6acb86" containerID="fe3c14b88650acd85b2c1589eee746e08761c6f651f8c1dae5d043d24fabb3fc" exitCode=0 Nov 03 17:48:39.570770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:39.570358 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" event={"ID":"2c3a650b-034a-494a-84f2-d91c5b6acb86","Type":"ContainerDied","Data":"fe3c14b88650acd85b2c1589eee746e08761c6f651f8c1dae5d043d24fabb3fc"} Nov 03 17:48:39.570770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:39.570432 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" event={"ID":"2c3a650b-034a-494a-84f2-d91c5b6acb86","Type":"ContainerStarted","Data":"4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37"} Nov 03 17:48:40.711631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.711605 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:40.767425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.767388 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\") pod \"2c3a650b-034a-494a-84f2-d91c5b6acb86\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " Nov 03 17:48:40.767425 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.767429 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\") pod \"2c3a650b-034a-494a-84f2-d91c5b6acb86\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " Nov 03 17:48:40.767735 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.767501 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\" (UniqueName: \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\") pod \"2c3a650b-034a-494a-84f2-d91c5b6acb86\" (UID: \"2c3a650b-034a-494a-84f2-d91c5b6acb86\") " Nov 03 17:48:40.767807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.767771 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca" pod "2c3a650b-034a-494a-84f2-d91c5b6acb86" (UID: "2c3a650b-034a-494a-84f2-d91c5b6acb86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:48:40.767920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.767898 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:40.769659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.769633 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca" pod "2c3a650b-034a-494a-84f2-d91c5b6acb86" (UID: "2c3a650b-034a-494a-84f2-d91c5b6acb86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:48:40.769791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.769666 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr" pod "2c3a650b-034a-494a-84f2-d91c5b6acb86" (UID: "2c3a650b-034a-494a-84f2-d91c5b6acb86"). InnerVolumeSpecName "kube-api-access-h68qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:48:40.868868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.868785 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c3a650b-034a-494a-84f2-d91c5b6acb86-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:40.868868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:40.868838 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68qr\" (UniqueName: \"kubernetes.io/projected/2c3a650b-034a-494a-84f2-d91c5b6acb86-kube-api-access-h68qr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:48:41.584159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:41.584121 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" event={"ID":"2c3a650b-034a-494a-84f2-d91c5b6acb86","Type":"ContainerDied","Data":"4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37"} Nov 03 17:48:41.584352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:41.584203 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ecb0f3863ae9279c8ec2ae5f6d50e65e0a5f9de7c5e0a9ac6dab4203cb98a37" Nov 03 17:48:41.584352 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:41.584135 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6" Nov 03 17:48:52.249087 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:48:52.249052 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:48:57.759980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.759942 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d"] Nov 03 17:48:57.760397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.760360 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2c3a650b-034a-494a-84f2-d91c5b6acb86" containerName="revoke-keys" Nov 03 17:48:57.760397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.760372 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3a650b-034a-494a-84f2-d91c5b6acb86" containerName="revoke-keys" Nov 03 17:48:57.760505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.760499 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3a650b-034a-494a-84f2-d91c5b6acb86" containerName="revoke-keys" Nov 03 17:48:57.763378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.763355 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.778565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.778539 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:48:57.778565 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.778562 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:48:57.791129 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.791089 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:48:57.791322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.791299 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:48:57.794205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.794173 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d"] Nov 03 17:48:57.846197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.846152 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.846197 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.846197 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.846660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.846613 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6nwf\" (UniqueName: \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.947811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.947773 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.947811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.947813 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.948072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.947882 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6nwf\" (UniqueName: \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.948494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.948433 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.950226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.950208 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:57.971742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:57.971717 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6nwf\" (UniqueName: \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\") pod \"maas-api-revoke-sub-2e1f5b5f-8qj2d\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:58.077096 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:58.077015 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:48:58.380265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:58.380232 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d"] Nov 03 17:48:58.381368 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:48:58.381338 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ca48a0_5031_480e_964a_1324ad2d7b27.slice/crio-8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292 WatchSource:0}: Error finding container 8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292: Status 404 returned error can't find the container with id 8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292 Nov 03 17:48:58.706772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:58.706740 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" event={"ID":"97ca48a0-5031-480e-964a-1324ad2d7b27","Type":"ContainerStarted","Data":"f23f4295e38211adf57dd3432118e94840e7201900aed6000a6ba50e039f9e5f"} Nov 03 17:48:58.706905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:58.706796 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" event={"ID":"97ca48a0-5031-480e-964a-1324ad2d7b27","Type":"ContainerStarted","Data":"8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292"} Nov 03 17:48:58.749684 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:58.749616 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" podStartSLOduration=1.749601494 podStartE2EDuration="1.749601494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:48:58.748587388 +0000 UTC m=+5634.081800957" watchObservedRunningTime="2026-11-03 17:48:58.749601494 +0000 UTC m=+5634.082815062" Nov 03 17:48:59.712503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:59.712464 6001 generic.go:356] "Generic (PLEG): container finished" podID="97ca48a0-5031-480e-964a-1324ad2d7b27" containerID="f23f4295e38211adf57dd3432118e94840e7201900aed6000a6ba50e039f9e5f" exitCode=0 Nov 03 17:48:59.712912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:48:59.712580 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" event={"ID":"97ca48a0-5031-480e-964a-1324ad2d7b27","Type":"ContainerDied","Data":"f23f4295e38211adf57dd3432118e94840e7201900aed6000a6ba50e039f9e5f"} Nov 03 17:49:00.857700 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.857673 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:49:00.982824 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.982792 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\" (UniqueName: \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\") pod \"97ca48a0-5031-480e-964a-1324ad2d7b27\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " Nov 03 17:49:00.983013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.982882 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\") pod \"97ca48a0-5031-480e-964a-1324ad2d7b27\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " Nov 03 17:49:00.983013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.982942 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\") pod \"97ca48a0-5031-480e-964a-1324ad2d7b27\" (UID: \"97ca48a0-5031-480e-964a-1324ad2d7b27\") " Nov 03 17:49:00.983260 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.983235 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca" pod "97ca48a0-5031-480e-964a-1324ad2d7b27" (UID: "97ca48a0-5031-480e-964a-1324ad2d7b27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:00.985094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.985067 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca" pod "97ca48a0-5031-480e-964a-1324ad2d7b27" (UID: "97ca48a0-5031-480e-964a-1324ad2d7b27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:00.985227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:00.985128 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf" pod "97ca48a0-5031-480e-964a-1324ad2d7b27" (UID: "97ca48a0-5031-480e-964a-1324ad2d7b27"). InnerVolumeSpecName "kube-api-access-t6nwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:01.083980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.083938 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6nwf\" (UniqueName: \"kubernetes.io/projected/97ca48a0-5031-480e-964a-1324ad2d7b27-kube-api-access-t6nwf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:01.083980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.083974 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:01.083980 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.083984 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97ca48a0-5031-480e-964a-1324ad2d7b27-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:01.724387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.724355 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" Nov 03 17:49:01.724614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.724354 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d" event={"ID":"97ca48a0-5031-480e-964a-1324ad2d7b27","Type":"ContainerDied","Data":"8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292"} Nov 03 17:49:01.724614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:01.724528 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8288cc4c171442c34e041a1b300e0423a0e84bb6ad72916bf094ab5eae040292" Nov 03 17:49:08.819543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.819505 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg"] Nov 03 17:49:08.820090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.819941 6001 cpu_manager.go:414] "containerMap: removing container" podUID="97ca48a0-5031-480e-964a-1324ad2d7b27" containerName="revoke-keys" Nov 03 17:49:08.820090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.819953 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ca48a0-5031-480e-964a-1324ad2d7b27" containerName="revoke-keys" Nov 03 17:49:08.820090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.820030 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ca48a0-5031-480e-964a-1324ad2d7b27" containerName="revoke-keys" Nov 03 17:49:08.823426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.823403 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:08.853918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.853885 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:49:08.853918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.853891 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:49:08.854223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.853920 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:49:08.854223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.854130 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:49:08.855954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.855927 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg"] Nov 03 17:49:08.966862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.966827 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:08.966862 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.966869 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:08.967094 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:08.966972 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww2dl\" (UniqueName: \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.068341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.068305 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww2dl\" (UniqueName: \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.068576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.068405 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.068576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.068427 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.069087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.069067 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.070963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.070916 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.092105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.092076 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww2dl\" (UniqueName: \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\") pod \"maas-api-revoke-sub-ec843018-l4qdg\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.137853 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.137818 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:09.439796 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.439764 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg"] Nov 03 17:49:09.440642 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:49:09.440608 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ba9f39_f4dc_451b_adff_78a65826dd28.slice/crio-863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012 WatchSource:0}: Error finding container 863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012: Status 404 returned error can't find the container with id 863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012 Nov 03 17:49:09.773379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.773342 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" event={"ID":"66ba9f39-f4dc-451b-adff-78a65826dd28","Type":"ContainerStarted","Data":"f9e2eb59f505f8ab4b21c6bc646031672146f27708d785973ffa314e21d88bd5"} Nov 03 17:49:09.773569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.773404 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" event={"ID":"66ba9f39-f4dc-451b-adff-78a65826dd28","Type":"ContainerStarted","Data":"863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012"} Nov 03 17:49:09.818838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:09.818784 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" podStartSLOduration=1.818770684 podStartE2EDuration="1.818770684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:09.817332536 +0000 UTC m=+5645.150546104" watchObservedRunningTime="2026-11-03 17:49:09.818770684 +0000 UTC m=+5645.151984244" Nov 03 17:49:10.780741 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:10.780703 6001 generic.go:356] "Generic (PLEG): container finished" podID="66ba9f39-f4dc-451b-adff-78a65826dd28" containerID="f9e2eb59f505f8ab4b21c6bc646031672146f27708d785973ffa314e21d88bd5" exitCode=0 Nov 03 17:49:10.781196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:10.780773 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" event={"ID":"66ba9f39-f4dc-451b-adff-78a65826dd28","Type":"ContainerDied","Data":"f9e2eb59f505f8ab4b21c6bc646031672146f27708d785973ffa314e21d88bd5"} Nov 03 17:49:11.947343 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:11.947309 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:12.103010 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.102909 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\") pod \"66ba9f39-f4dc-451b-adff-78a65826dd28\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " Nov 03 17:49:12.103206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.103116 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\" (UniqueName: \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\") pod \"66ba9f39-f4dc-451b-adff-78a65826dd28\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " Nov 03 17:49:12.103206 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.103149 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\") pod \"66ba9f39-f4dc-451b-adff-78a65826dd28\" (UID: \"66ba9f39-f4dc-451b-adff-78a65826dd28\") " Nov 03 17:49:12.103828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.103799 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca" pod "66ba9f39-f4dc-451b-adff-78a65826dd28" (UID: "66ba9f39-f4dc-451b-adff-78a65826dd28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:12.105526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.105499 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl" pod "66ba9f39-f4dc-451b-adff-78a65826dd28" (UID: "66ba9f39-f4dc-451b-adff-78a65826dd28"). InnerVolumeSpecName "kube-api-access-ww2dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:12.106023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.105998 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca" pod "66ba9f39-f4dc-451b-adff-78a65826dd28" (UID: "66ba9f39-f4dc-451b-adff-78a65826dd28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:12.205262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.205210 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww2dl\" (UniqueName: \"kubernetes.io/projected/66ba9f39-f4dc-451b-adff-78a65826dd28-kube-api-access-ww2dl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:12.205262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.205259 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:12.205615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.205277 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66ba9f39-f4dc-451b-adff-78a65826dd28-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:12.796942 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.796909 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" Nov 03 17:49:12.797237 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.797198 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg" event={"ID":"66ba9f39-f4dc-451b-adff-78a65826dd28","Type":"ContainerDied","Data":"863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012"} Nov 03 17:49:12.797378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:12.797267 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863641f6b7da86c03475c0e612e4e00799bae48bff3f99bdb2f52d5283757012" Nov 03 17:49:22.250035 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:49:22.249998 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:49:36.472427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.472387 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh"] Nov 03 17:49:36.473165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.473144 6001 cpu_manager.go:414] "containerMap: removing container" podUID="66ba9f39-f4dc-451b-adff-78a65826dd28" containerName="revoke-keys" Nov 03 17:49:36.473233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.473167 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ba9f39-f4dc-451b-adff-78a65826dd28" containerName="revoke-keys" Nov 03 17:49:36.473306 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.473294 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ba9f39-f4dc-451b-adff-78a65826dd28" containerName="revoke-keys" Nov 03 17:49:36.477174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.477149 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.503514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.503483 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh"] Nov 03 17:49:36.513018 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.512985 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:49:36.513198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.513071 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:49:36.513198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.513096 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:49:36.513198 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.513131 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:49:36.597136 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.597100 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.597335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.597169 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.597335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.597312 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4cqn\" (UniqueName: \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.698537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.698503 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.698779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.698587 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.698779 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.698727 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4cqn\" (UniqueName: \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.699329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.699300 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.701121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.701096 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.741597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.741522 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4cqn\" (UniqueName: \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\") pod \"maas-api-revoke-sub-046e1343-mvxgh\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:36.791904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:36.791871 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:37.095778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:37.095744 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh"] Nov 03 17:49:37.097280 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:49:37.097248 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c56f89_5e8b_4ef5_a527_1dab9878db1b.slice/crio-694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34 WatchSource:0}: Error finding container 694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34: Status 404 returned error can't find the container with id 694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34 Nov 03 17:49:37.957382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:37.957341 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" event={"ID":"40c56f89-5e8b-4ef5-a527-1dab9878db1b","Type":"ContainerStarted","Data":"a7f16bc2c963ee3456bce0a118977b5b98b085b7fa1caceed5a89d23ecbe4e79"} Nov 03 17:49:37.957761 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:37.957411 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" event={"ID":"40c56f89-5e8b-4ef5-a527-1dab9878db1b","Type":"ContainerStarted","Data":"694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34"} Nov 03 17:49:38.964874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:38.964837 6001 generic.go:356] "Generic (PLEG): container finished" podID="40c56f89-5e8b-4ef5-a527-1dab9878db1b" containerID="a7f16bc2c963ee3456bce0a118977b5b98b085b7fa1caceed5a89d23ecbe4e79" exitCode=0 Nov 03 17:49:38.965327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:38.964878 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" event={"ID":"40c56f89-5e8b-4ef5-a527-1dab9878db1b","Type":"ContainerDied","Data":"a7f16bc2c963ee3456bce0a118977b5b98b085b7fa1caceed5a89d23ecbe4e79"} Nov 03 17:49:39.103391 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.103364 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:39.227758 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.227724 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\") pod \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " Nov 03 17:49:39.227917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.227785 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\") pod \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " Nov 03 17:49:39.227917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.227816 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\" (UniqueName: \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\") pod \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\" (UID: \"40c56f89-5e8b-4ef5-a527-1dab9878db1b\") " Nov 03 17:49:39.228221 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.228192 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca" pod "40c56f89-5e8b-4ef5-a527-1dab9878db1b" (UID: "40c56f89-5e8b-4ef5-a527-1dab9878db1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:39.229998 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.229974 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca" pod "40c56f89-5e8b-4ef5-a527-1dab9878db1b" (UID: "40c56f89-5e8b-4ef5-a527-1dab9878db1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:39.230133 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.230046 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn" pod "40c56f89-5e8b-4ef5-a527-1dab9878db1b" (UID: "40c56f89-5e8b-4ef5-a527-1dab9878db1b"). InnerVolumeSpecName "kube-api-access-w4cqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:39.329140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.329102 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:39.329140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.329137 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c56f89-5e8b-4ef5-a527-1dab9878db1b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:39.329140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.329147 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4cqn\" (UniqueName: \"kubernetes.io/projected/40c56f89-5e8b-4ef5-a527-1dab9878db1b-kube-api-access-w4cqn\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:39.971904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.971853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" event={"ID":"40c56f89-5e8b-4ef5-a527-1dab9878db1b","Type":"ContainerDied","Data":"694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34"} Nov 03 17:49:39.971904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.971885 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh" Nov 03 17:49:39.972400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:39.971921 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="694a30922fbd4482cf025b7aeb80711ce7922b96b7d9eb22d43918bf7e271d34" Nov 03 17:49:46.820290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.820200 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j"] Nov 03 17:49:46.820762 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.820742 6001 cpu_manager.go:414] "containerMap: removing container" podUID="40c56f89-5e8b-4ef5-a527-1dab9878db1b" containerName="revoke-keys" Nov 03 17:49:46.820834 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.820763 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c56f89-5e8b-4ef5-a527-1dab9878db1b" containerName="revoke-keys" Nov 03 17:49:46.820897 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.820847 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c56f89-5e8b-4ef5-a527-1dab9878db1b" containerName="revoke-keys" Nov 03 17:49:46.824263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.824246 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:46.842513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.842481 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:49:46.845552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.845517 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j"] Nov 03 17:49:46.855099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.855074 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:49:46.905990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.905956 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:49:46.905990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.905956 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:49:46.913490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.913457 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:46.913644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.913520 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:46.913644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:46.913615 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtcht\" (UniqueName: \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.014553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.014504 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.014764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.014592 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.014764 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.014621 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtcht\" (UniqueName: \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.015315 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.015291 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.017098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.017076 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.038882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.038850 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtcht\" (UniqueName: \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\") pod \"maas-api-revoke-sub-d3fba056-pp47j\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.137951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.137856 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:47.444140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:47.444102 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j"] Nov 03 17:49:47.444820 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:49:47.444791 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0170d915_8c7a_4e77_bf5c_08aa879c6a5b.slice/crio-6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4 WatchSource:0}: Error finding container 6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4: Status 404 returned error can't find the container with id 6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4 Nov 03 17:49:48.018671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:48.018574 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" event={"ID":"0170d915-8c7a-4e77-bf5c-08aa879c6a5b","Type":"ContainerStarted","Data":"1c9c2c502842c2c9519f23fe0be7bcd23d23ab0be94ffcb3d2906893312951aa"} Nov 03 17:49:48.018671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:48.018647 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" event={"ID":"0170d915-8c7a-4e77-bf5c-08aa879c6a5b","Type":"ContainerStarted","Data":"6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4"} Nov 03 17:49:48.060886 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:48.060828 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" podStartSLOduration=2.06081432 podStartE2EDuration="2.06081432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:48.059845646 +0000 UTC m=+5683.393059214" watchObservedRunningTime="2026-11-03 17:49:48.06081432 +0000 UTC m=+5683.394027888" Nov 03 17:49:49.029630 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:49.029591 6001 generic.go:356] "Generic (PLEG): container finished" podID="0170d915-8c7a-4e77-bf5c-08aa879c6a5b" containerID="1c9c2c502842c2c9519f23fe0be7bcd23d23ab0be94ffcb3d2906893312951aa" exitCode=0 Nov 03 17:49:49.030368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:49.030338 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" event={"ID":"0170d915-8c7a-4e77-bf5c-08aa879c6a5b","Type":"ContainerDied","Data":"1c9c2c502842c2c9519f23fe0be7bcd23d23ab0be94ffcb3d2906893312951aa"} Nov 03 17:49:50.439686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.439642 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:50.609532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.608927 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\") pod \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " Nov 03 17:49:50.609532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.608988 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\") pod \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " Nov 03 17:49:50.609532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.609136 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\" (UniqueName: \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\") pod \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\" (UID: \"0170d915-8c7a-4e77-bf5c-08aa879c6a5b\") " Nov 03 17:49:50.617428 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.615742 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca" pod "0170d915-8c7a-4e77-bf5c-08aa879c6a5b" (UID: "0170d915-8c7a-4e77-bf5c-08aa879c6a5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:49:50.619522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.619483 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht" pod "0170d915-8c7a-4e77-bf5c-08aa879c6a5b" (UID: "0170d915-8c7a-4e77-bf5c-08aa879c6a5b"). InnerVolumeSpecName "kube-api-access-dtcht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:49:50.620294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.620236 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca" pod "0170d915-8c7a-4e77-bf5c-08aa879c6a5b" (UID: "0170d915-8c7a-4e77-bf5c-08aa879c6a5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:49:50.711517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.711405 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:50.711517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.711462 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:50.711517 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:50.711478 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtcht\" (UniqueName: \"kubernetes.io/projected/0170d915-8c7a-4e77-bf5c-08aa879c6a5b-kube-api-access-dtcht\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:49:51.042432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:51.042404 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" Nov 03 17:49:51.042432 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:51.042454 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j" event={"ID":"0170d915-8c7a-4e77-bf5c-08aa879c6a5b","Type":"ContainerDied","Data":"6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4"} Nov 03 17:49:51.042717 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:49:51.042506 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f05811c3b3770639563fe486121e232b0d833af0f8124b53b0c379953021eb4" Nov 03 17:49:52.251214 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:49:52.251173 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:50:05.933874 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:05.933843 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:50:06.052811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:06.052783 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:50:08.219703 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.219668 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk"] Nov 03 17:50:08.220217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.220108 6001 cpu_manager.go:414] "containerMap: removing container" podUID="0170d915-8c7a-4e77-bf5c-08aa879c6a5b" containerName="revoke-keys" Nov 03 17:50:08.220217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.220121 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0170d915-8c7a-4e77-bf5c-08aa879c6a5b" containerName="revoke-keys" Nov 03 17:50:08.220316 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.220225 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="0170d915-8c7a-4e77-bf5c-08aa879c6a5b" containerName="revoke-keys" Nov 03 17:50:08.223611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.223591 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.238952 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.238922 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:50:08.254543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.251689 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:50:08.254543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.251716 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:50:08.254543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.251722 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:50:08.260615 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.260571 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk"] Nov 03 17:50:08.313865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.313818 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.314070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.313874 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.314070 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.313935 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdf4\" (UniqueName: \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.414919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.414871 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.414919 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.414929 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.415183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.414957 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdf4\" (UniqueName: \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.415756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.415730 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.417434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.417410 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.439041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.439005 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdf4\" (UniqueName: \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\") pod \"maas-api-revoke-sub-ae594979-8dksk\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.537248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.537201 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:08.842562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:08.842530 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk"] Nov 03 17:50:08.843493 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:50:08.843462 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea23e79e_3f23_46c5_8ab5_6e226e309a4f.slice/crio-6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60 WatchSource:0}: Error finding container 6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60: Status 404 returned error can't find the container with id 6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60 Nov 03 17:50:09.154574 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:09.154541 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" event={"ID":"ea23e79e-3f23-46c5-8ab5-6e226e309a4f","Type":"ContainerStarted","Data":"6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60"} Nov 03 17:50:10.161539 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:10.161504 6001 generic.go:356] "Generic (PLEG): container finished" podID="ea23e79e-3f23-46c5-8ab5-6e226e309a4f" containerID="6d40e1bd25e3e33934872c53d19f47f1d8f5d3b041615b11f5229d793a44b56e" exitCode=0 Nov 03 17:50:10.161977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:10.161598 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" event={"ID":"ea23e79e-3f23-46c5-8ab5-6e226e309a4f","Type":"ContainerDied","Data":"6d40e1bd25e3e33934872c53d19f47f1d8f5d3b041615b11f5229d793a44b56e"} Nov 03 17:50:11.311055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.311023 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:11.347753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.347704 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\" (UniqueName: \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\") pod \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " Nov 03 17:50:11.347962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.347792 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\") pod \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " Nov 03 17:50:11.347962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.347828 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\") pod \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\" (UID: \"ea23e79e-3f23-46c5-8ab5-6e226e309a4f\") " Nov 03 17:50:11.348261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.348239 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca" pod "ea23e79e-3f23-46c5-8ab5-6e226e309a4f" (UID: "ea23e79e-3f23-46c5-8ab5-6e226e309a4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:11.350046 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.350020 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca" pod "ea23e79e-3f23-46c5-8ab5-6e226e309a4f" (UID: "ea23e79e-3f23-46c5-8ab5-6e226e309a4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:50:11.350185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.350100 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4" pod "ea23e79e-3f23-46c5-8ab5-6e226e309a4f" (UID: "ea23e79e-3f23-46c5-8ab5-6e226e309a4f"). InnerVolumeSpecName "kube-api-access-rpdf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:11.449767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.449667 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdf4\" (UniqueName: \"kubernetes.io/projected/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-kube-api-access-rpdf4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:11.449767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.449705 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:11.449767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:11.449714 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea23e79e-3f23-46c5-8ab5-6e226e309a4f-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:12.173327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:12.173295 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" Nov 03 17:50:12.173560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:12.173321 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk" event={"ID":"ea23e79e-3f23-46c5-8ab5-6e226e309a4f","Type":"ContainerDied","Data":"6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60"} Nov 03 17:50:12.173560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:12.173377 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f90618f683da8598a50b3e711419738453d24593f0d343411b1e83fd3918a60" Nov 03 17:50:14.787929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:14.787899 6001 scope.go:122] "RemoveContainer" containerID="2a0a3c5885313cac804a0c120f8d4a2a0e2a416c44a57679c5babb5ccf2be2b1" Nov 03 17:50:22.259897 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:50:22.259854 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:50:25.337078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.337042 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59"] Nov 03 17:50:25.337680 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.337658 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ea23e79e-3f23-46c5-8ab5-6e226e309a4f" containerName="revoke-keys" Nov 03 17:50:25.337775 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.337682 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea23e79e-3f23-46c5-8ab5-6e226e309a4f" containerName="revoke-keys" Nov 03 17:50:25.337876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.337862 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea23e79e-3f23-46c5-8ab5-6e226e309a4f" containerName="revoke-keys" Nov 03 17:50:25.341176 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.341144 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.369016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.368986 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:50:25.369257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.368986 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:50:25.369370 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.369116 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:50:25.369434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.369137 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:50:25.372264 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.372234 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59"] Nov 03 17:50:25.523742 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.523702 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.523977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.523756 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.523977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.523865 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t46mz\" (UniqueName: \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.625397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.625303 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.625397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.625353 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.625397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.625380 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t46mz\" (UniqueName: \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.626099 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.626071 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.627732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.627701 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.649429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.649400 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t46mz\" (UniqueName: \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\") pod \"maas-api-revoke-sub-a4aef7d7-5nf59\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.658245 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.658218 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:25.959075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:25.959044 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59"] Nov 03 17:50:25.959942 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:50:25.959914 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f26b91_190b_4a03_af4d_f1b3a846c3e4.slice/crio-f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d WatchSource:0}: Error finding container f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d: Status 404 returned error can't find the container with id f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d Nov 03 17:50:26.250248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:26.250214 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" event={"ID":"24f26b91-190b-4a03-af4d-f1b3a846c3e4","Type":"ContainerStarted","Data":"f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d"} Nov 03 17:50:27.257916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:27.257878 6001 generic.go:356] "Generic (PLEG): container finished" podID="24f26b91-190b-4a03-af4d-f1b3a846c3e4" containerID="5bcb40ec16b1236e67ef44c359e7afae79c6ce2c24645ce8c3da8afcd0a21146" exitCode=0 Nov 03 17:50:27.258435 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:27.257965 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" event={"ID":"24f26b91-190b-4a03-af4d-f1b3a846c3e4","Type":"ContainerDied","Data":"5bcb40ec16b1236e67ef44c359e7afae79c6ce2c24645ce8c3da8afcd0a21146"} Nov 03 17:50:28.402223 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.402194 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:28.555990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.555888 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\" (UniqueName: \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\") pod \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " Nov 03 17:50:28.556174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.556086 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\") pod \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " Nov 03 17:50:28.556174 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.556110 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\") pod \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\" (UID: \"24f26b91-190b-4a03-af4d-f1b3a846c3e4\") " Nov 03 17:50:28.556537 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.556510 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca" pod "24f26b91-190b-4a03-af4d-f1b3a846c3e4" (UID: "24f26b91-190b-4a03-af4d-f1b3a846c3e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:28.558109 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.558086 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz" pod "24f26b91-190b-4a03-af4d-f1b3a846c3e4" (UID: "24f26b91-190b-4a03-af4d-f1b3a846c3e4"). InnerVolumeSpecName "kube-api-access-t46mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:28.558187 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.558165 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca" pod "24f26b91-190b-4a03-af4d-f1b3a846c3e4" (UID: "24f26b91-190b-4a03-af4d-f1b3a846c3e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:50:28.657557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.657511 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:28.657557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.657551 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24f26b91-190b-4a03-af4d-f1b3a846c3e4-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:28.657557 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:28.657566 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t46mz\" (UniqueName: \"kubernetes.io/projected/24f26b91-190b-4a03-af4d-f1b3a846c3e4-kube-api-access-t46mz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:29.277003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:29.276970 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" Nov 03 17:50:29.277215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:29.277084 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59" event={"ID":"24f26b91-190b-4a03-af4d-f1b3a846c3e4","Type":"ContainerDied","Data":"f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d"} Nov 03 17:50:29.277215 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:29.277132 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b63eba7e7de34d7a98d3d75c39a4d5f611636a81bc17a0d39dfd6e46df645d" Nov 03 17:50:31.032086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:31.032047 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8"] Nov 03 17:50:31.045767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:31.045729 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-8b6s8"] Nov 03 17:50:32.788440 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:32.788400 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a5d742-40dd-497a-9e86-cc73ff94b32c" path="/var/lib/kubelet/pods/a7a5d742-40dd-497a-9e86-cc73ff94b32c/volumes" Nov 03 17:50:42.141658 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.141581 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2"] Nov 03 17:50:42.142360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.142232 6001 cpu_manager.go:414] "containerMap: removing container" podUID="24f26b91-190b-4a03-af4d-f1b3a846c3e4" containerName="revoke-keys" Nov 03 17:50:42.142360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.142251 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f26b91-190b-4a03-af4d-f1b3a846c3e4" containerName="revoke-keys" Nov 03 17:50:42.142513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.142395 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f26b91-190b-4a03-af4d-f1b3a846c3e4" containerName="revoke-keys" Nov 03 17:50:42.150140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.150108 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.173192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.173145 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2"] Nov 03 17:50:42.183458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.182604 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:50:42.183458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.182663 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:50:42.183458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.182748 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:50:42.183458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.183393 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:50:42.206802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.202122 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.206802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.202185 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.206802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.202341 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmpm\" (UniqueName: \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.312226 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.312168 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.313280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.312562 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.313280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.312764 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmpm\" (UniqueName: \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.313959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.313927 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.316250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.316224 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.337531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.337500 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmpm\" (UniqueName: \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\") pod \"maas-api-revoke-sub-79805ca9-mg8c2\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:42.513704 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:42.513656 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:43.022811 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:43.022757 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2"] Nov 03 17:50:43.405384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:43.405286 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" event={"ID":"9ece5404-2365-4aaa-96de-5ab03e3e1b81","Type":"ContainerStarted","Data":"6b8c796044090188823e51c0654f75d72dedcc83262bee8d48036af8fffd5412"} Nov 03 17:50:44.413439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:44.413345 6001 generic.go:356] "Generic (PLEG): container finished" podID="9ece5404-2365-4aaa-96de-5ab03e3e1b81" containerID="f37d2c0ae91aabdc6f8adff873d33082fe2bacc68cb640192265c5159bfb60e7" exitCode=0 Nov 03 17:50:44.413847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:44.413438 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" event={"ID":"9ece5404-2365-4aaa-96de-5ab03e3e1b81","Type":"ContainerDied","Data":"f37d2c0ae91aabdc6f8adff873d33082fe2bacc68cb640192265c5159bfb60e7"} Nov 03 17:50:45.559642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.559609 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:45.660846 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.660812 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\" (UniqueName: \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\") pod \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " Nov 03 17:50:45.661039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.660866 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\") pod \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " Nov 03 17:50:45.661039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.660965 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\") pod \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\" (UID: \"9ece5404-2365-4aaa-96de-5ab03e3e1b81\") " Nov 03 17:50:45.661358 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.661324 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca" pod "9ece5404-2365-4aaa-96de-5ab03e3e1b81" (UID: "9ece5404-2365-4aaa-96de-5ab03e3e1b81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:45.663035 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.663005 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca" pod "9ece5404-2365-4aaa-96de-5ab03e3e1b81" (UID: "9ece5404-2365-4aaa-96de-5ab03e3e1b81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:50:45.663158 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.663134 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm" pod "9ece5404-2365-4aaa-96de-5ab03e3e1b81" (UID: "9ece5404-2365-4aaa-96de-5ab03e3e1b81"). InnerVolumeSpecName "kube-api-access-cbmpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:45.761912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.761876 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:45.761912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.761908 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmpm\" (UniqueName: \"kubernetes.io/projected/9ece5404-2365-4aaa-96de-5ab03e3e1b81-kube-api-access-cbmpm\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:45.761912 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:45.761919 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ece5404-2365-4aaa-96de-5ab03e3e1b81-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:46.426864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:46.426822 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" Nov 03 17:50:46.426864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:46.426856 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2" event={"ID":"9ece5404-2365-4aaa-96de-5ab03e3e1b81","Type":"ContainerDied","Data":"6b8c796044090188823e51c0654f75d72dedcc83262bee8d48036af8fffd5412"} Nov 03 17:50:46.427107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:46.426910 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b8c796044090188823e51c0654f75d72dedcc83262bee8d48036af8fffd5412" Nov 03 17:50:49.031047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:49.031009 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5"] Nov 03 17:50:49.043325 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:49.043290 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe064636-lq7j5"] Nov 03 17:50:50.788590 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:50.788553 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ccdbaf-0139-43dc-970d-cafc6b699911" path="/var/lib/kubelet/pods/c9ccdbaf-0139-43dc-970d-cafc6b699911/volumes" Nov 03 17:50:52.249244 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:50:52.249200 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:50:55.807387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.804873 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx"] Nov 03 17:50:55.807387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.805594 6001 cpu_manager.go:414] "containerMap: removing container" podUID="9ece5404-2365-4aaa-96de-5ab03e3e1b81" containerName="revoke-keys" Nov 03 17:50:55.807387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.805615 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ece5404-2365-4aaa-96de-5ab03e3e1b81" containerName="revoke-keys" Nov 03 17:50:55.807387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.805779 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ece5404-2365-4aaa-96de-5ab03e3e1b81" containerName="revoke-keys" Nov 03 17:50:55.810648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.809889 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.828456 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.828301 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx"] Nov 03 17:50:55.838125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.838089 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:50:55.838125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.838115 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:50:55.838390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.838142 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:50:55.838390 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.838185 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:50:55.879295 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.879256 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.879726 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.879689 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l26w\" (UniqueName: \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.879983 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.879964 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.989128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.989078 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.990053 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.990024 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l26w\" (UniqueName: \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:55.993211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:55.993186 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:56.001268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:56.000283 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:56.001471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:56.001141 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:56.021876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:56.021838 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l26w\" (UniqueName: \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\") pod \"maas-api-revoke-sub-0b7e8edb-7wpxx\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:56.134991 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:56.134892 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:56.591507 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:56.584970 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx"] Nov 03 17:50:56.617011 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:50:56.616967 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3f3fd3_394b_4202_8540_ceafaf235a73.slice/crio-3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f WatchSource:0}: Error finding container 3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f: Status 404 returned error can't find the container with id 3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f Nov 03 17:50:57.536326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:57.535748 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" event={"ID":"ee3f3fd3-394b-4202-8540-ceafaf235a73","Type":"ContainerStarted","Data":"7090bda958a8ff9d81ee5dff96cb9c9f8d2054d73e9d8fe7cfc6e3780f7ed6ae"} Nov 03 17:50:57.536326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:57.535806 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" event={"ID":"ee3f3fd3-394b-4202-8540-ceafaf235a73","Type":"ContainerStarted","Data":"3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f"} Nov 03 17:50:57.578835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:57.578765 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" podStartSLOduration=2.578744044 podStartE2EDuration="2.578744044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:57.578181557 +0000 UTC m=+5752.911395126" watchObservedRunningTime="2026-11-03 17:50:57.578744044 +0000 UTC m=+5752.911957614" Nov 03 17:50:58.543014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:58.542979 6001 generic.go:356] "Generic (PLEG): container finished" podID="ee3f3fd3-394b-4202-8540-ceafaf235a73" containerID="7090bda958a8ff9d81ee5dff96cb9c9f8d2054d73e9d8fe7cfc6e3780f7ed6ae" exitCode=0 Nov 03 17:50:58.543497 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:58.543095 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" event={"ID":"ee3f3fd3-394b-4202-8540-ceafaf235a73","Type":"ContainerDied","Data":"7090bda958a8ff9d81ee5dff96cb9c9f8d2054d73e9d8fe7cfc6e3780f7ed6ae"} Nov 03 17:50:59.684346 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.684315 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:50:59.762217 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.762187 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\") pod \"ee3f3fd3-394b-4202-8540-ceafaf235a73\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " Nov 03 17:50:59.762415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.762298 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\" (UniqueName: \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\") pod \"ee3f3fd3-394b-4202-8540-ceafaf235a73\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " Nov 03 17:50:59.762415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.762328 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\") pod \"ee3f3fd3-394b-4202-8540-ceafaf235a73\" (UID: \"ee3f3fd3-394b-4202-8540-ceafaf235a73\") " Nov 03 17:50:59.762746 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.762721 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca" pod "ee3f3fd3-394b-4202-8540-ceafaf235a73" (UID: "ee3f3fd3-394b-4202-8540-ceafaf235a73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:59.764305 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.764282 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca" pod "ee3f3fd3-394b-4202-8540-ceafaf235a73" (UID: "ee3f3fd3-394b-4202-8540-ceafaf235a73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:50:59.764434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.764416 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w" pod "ee3f3fd3-394b-4202-8540-ceafaf235a73" (UID: "ee3f3fd3-394b-4202-8540-ceafaf235a73"). InnerVolumeSpecName "kube-api-access-5l26w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:59.863666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.863580 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:59.863666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.863620 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l26w\" (UniqueName: \"kubernetes.io/projected/ee3f3fd3-394b-4202-8540-ceafaf235a73-kube-api-access-5l26w\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:50:59.863666 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:50:59.863635 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3f3fd3-394b-4202-8540-ceafaf235a73-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:00.554831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:00.554800 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" Nov 03 17:51:00.555023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:00.554805 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx" event={"ID":"ee3f3fd3-394b-4202-8540-ceafaf235a73","Type":"ContainerDied","Data":"3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f"} Nov 03 17:51:00.555023 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:00.554925 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3961c571edb921834b446c6b07af9403b4228c3f949687a05bc9dd26bbf4de9f" Nov 03 17:51:06.503760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.503722 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm"] Nov 03 17:51:06.504319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.504129 6001 cpu_manager.go:414] "containerMap: removing container" podUID="ee3f3fd3-394b-4202-8540-ceafaf235a73" containerName="revoke-keys" Nov 03 17:51:06.504319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.504141 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3f3fd3-394b-4202-8540-ceafaf235a73" containerName="revoke-keys" Nov 03 17:51:06.504319 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.504251 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3f3fd3-394b-4202-8540-ceafaf235a73" containerName="revoke-keys" Nov 03 17:51:06.507601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.507579 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.523067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.523033 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:51:06.535636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.535589 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:51:06.535636 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.535610 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:51:06.535958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.535711 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:51:06.537911 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.537882 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm"] Nov 03 17:51:06.631196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.631154 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2df\" (UniqueName: \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.631196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.631202 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.631493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.631317 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.732267 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.732231 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2df\" (UniqueName: \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.732515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.732280 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.732515 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.732356 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.732930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.732902 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.734843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.734820 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.756873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.756803 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2df\" (UniqueName: \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\") pod \"maas-api-revoke-sub-3d4156f1-57jdm\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:06.821097 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:06.821056 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:07.122268 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:07.122243 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm"] Nov 03 17:51:07.123185 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:51:07.123157 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f70f07_312a_4ed4_98dc_a15d4b69224b.slice/crio-9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d WatchSource:0}: Error finding container 9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d: Status 404 returned error can't find the container with id 9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d Nov 03 17:51:07.605155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:07.605112 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" event={"ID":"22f70f07-312a-4ed4-98dc-a15d4b69224b","Type":"ContainerStarted","Data":"cfa015f5beb8881e6dedcd41afb72cc45fa6f377b1bb3e775a35560dcf37900f"} Nov 03 17:51:07.605640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:07.605200 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" event={"ID":"22f70f07-312a-4ed4-98dc-a15d4b69224b","Type":"ContainerStarted","Data":"9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d"} Nov 03 17:51:07.648484 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:07.648410 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" podStartSLOduration=1.6483961360000001 podStartE2EDuration="1.648396136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:07.646960227 +0000 UTC m=+5762.980173796" watchObservedRunningTime="2026-11-03 17:51:07.648396136 +0000 UTC m=+5762.981609702" Nov 03 17:51:08.611371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:08.611338 6001 generic.go:356] "Generic (PLEG): container finished" podID="22f70f07-312a-4ed4-98dc-a15d4b69224b" containerID="cfa015f5beb8881e6dedcd41afb72cc45fa6f377b1bb3e775a35560dcf37900f" exitCode=0 Nov 03 17:51:08.611812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:08.611406 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" event={"ID":"22f70f07-312a-4ed4-98dc-a15d4b69224b","Type":"ContainerDied","Data":"cfa015f5beb8881e6dedcd41afb72cc45fa6f377b1bb3e775a35560dcf37900f"} Nov 03 17:51:09.760635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.760605 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:09.862733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.862697 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\" (UniqueName: \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\") pod \"22f70f07-312a-4ed4-98dc-a15d4b69224b\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " Nov 03 17:51:09.862959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.862771 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\") pod \"22f70f07-312a-4ed4-98dc-a15d4b69224b\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " Nov 03 17:51:09.862959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.862802 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\") pod \"22f70f07-312a-4ed4-98dc-a15d4b69224b\" (UID: \"22f70f07-312a-4ed4-98dc-a15d4b69224b\") " Nov 03 17:51:09.863238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.863206 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca" pod "22f70f07-312a-4ed4-98dc-a15d4b69224b" (UID: "22f70f07-312a-4ed4-98dc-a15d4b69224b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:51:09.865017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.864986 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca" pod "22f70f07-312a-4ed4-98dc-a15d4b69224b" (UID: "22f70f07-312a-4ed4-98dc-a15d4b69224b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:51:09.865017 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.865006 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df" pod "22f70f07-312a-4ed4-98dc-a15d4b69224b" (UID: "22f70f07-312a-4ed4-98dc-a15d4b69224b"). InnerVolumeSpecName "kube-api-access-mk2df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:51:09.964249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.964208 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:09.964249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.964239 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22f70f07-312a-4ed4-98dc-a15d4b69224b-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:09.964249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:09.964249 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2df\" (UniqueName: \"kubernetes.io/projected/22f70f07-312a-4ed4-98dc-a15d4b69224b-kube-api-access-mk2df\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:10.626809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:10.626777 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" Nov 03 17:51:10.627004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:10.626806 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm" event={"ID":"22f70f07-312a-4ed4-98dc-a15d4b69224b","Type":"ContainerDied","Data":"9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d"} Nov 03 17:51:10.627004 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:10.626855 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9234185a8e1540a629e0b4c6d9a5ec75aab8e41534324fb30d89eaf189b0f50d" Nov 03 17:51:14.809820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:14.809789 6001 scope.go:122] "RemoveContainer" containerID="243324e12297a09cb01b151558daa74f49edc1c1ff89c784855fb2627d689e0a" Nov 03 17:51:22.279408 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:51:22.279148 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:51:24.535838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.535801 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm"] Nov 03 17:51:24.549014 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.548983 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-57jdm"] Nov 03 17:51:24.585220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.585190 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5"] Nov 03 17:51:24.585753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.585729 6001 cpu_manager.go:414] "containerMap: removing container" podUID="22f70f07-312a-4ed4-98dc-a15d4b69224b" containerName="revoke-keys" Nov 03 17:51:24.585753 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.585749 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22f70f07-312a-4ed4-98dc-a15d4b69224b" containerName="revoke-keys" Nov 03 17:51:24.585925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.585856 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="22f70f07-312a-4ed4-98dc-a15d4b69224b" containerName="revoke-keys" Nov 03 17:51:24.588643 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.588622 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.603849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.603826 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:51:24.603849 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.603839 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:51:24.604067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.603943 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:51:24.604067 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.604008 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:51:24.613373 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.613345 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5"] Nov 03 17:51:24.695739 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.695698 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.695957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.695835 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.695957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.695866 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9p7\" (UniqueName: \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.788153 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.788078 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22f70f07-312a-4ed4-98dc-a15d4b69224b" path="/var/lib/kubelet/pods/22f70f07-312a-4ed4-98dc-a15d4b69224b/volumes" Nov 03 17:51:24.797254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.797230 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.797341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.797294 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.797341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.797315 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9p7\" (UniqueName: \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.797875 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.797855 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.799744 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.799721 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.821436 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.821399 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9p7\" (UniqueName: \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\") pod \"maas-api-revoke-sub-3d4156f1-lwvw5\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:24.901627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:24.901588 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:25.202062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:25.202031 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5"] Nov 03 17:51:25.203258 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:51:25.203235 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3536bf1_150e_463b_a4f5_a5d3eff8fe63.slice/crio-94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4 WatchSource:0}: Error finding container 94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4: Status 404 returned error can't find the container with id 94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4 Nov 03 17:51:25.753893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:25.753853 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" event={"ID":"b3536bf1-150e-463b-a4f5-a5d3eff8fe63","Type":"ContainerStarted","Data":"42990cf3cf8ab74e0cfa9344dd16247eba211fb80204fc4d5ed700ca861090fd"} Nov 03 17:51:25.754365 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:25.753914 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" event={"ID":"b3536bf1-150e-463b-a4f5-a5d3eff8fe63","Type":"ContainerStarted","Data":"94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4"} Nov 03 17:51:25.796832 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:25.796773 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" podStartSLOduration=1.796760218 podStartE2EDuration="1.796760218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:25.796082482 +0000 UTC m=+5781.129296050" watchObservedRunningTime="2026-11-03 17:51:25.796760218 +0000 UTC m=+5781.129973787" Nov 03 17:51:26.761016 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:26.760981 6001 generic.go:356] "Generic (PLEG): container finished" podID="b3536bf1-150e-463b-a4f5-a5d3eff8fe63" containerID="42990cf3cf8ab74e0cfa9344dd16247eba211fb80204fc4d5ed700ca861090fd" exitCode=0 Nov 03 17:51:26.761427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:26.761058 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" event={"ID":"b3536bf1-150e-463b-a4f5-a5d3eff8fe63","Type":"ContainerDied","Data":"42990cf3cf8ab74e0cfa9344dd16247eba211fb80204fc4d5ed700ca861090fd"} Nov 03 17:51:27.909081 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:27.909056 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:28.035011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.034921 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\" (UniqueName: \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\") pod \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " Nov 03 17:51:28.035011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.035008 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\") pod \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " Nov 03 17:51:28.035252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.035131 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\") pod \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\" (UID: \"b3536bf1-150e-463b-a4f5-a5d3eff8fe63\") " Nov 03 17:51:28.035364 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.035339 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca" pod "b3536bf1-150e-463b-a4f5-a5d3eff8fe63" (UID: "b3536bf1-150e-463b-a4f5-a5d3eff8fe63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:51:28.037118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.037090 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7" pod "b3536bf1-150e-463b-a4f5-a5d3eff8fe63" (UID: "b3536bf1-150e-463b-a4f5-a5d3eff8fe63"). InnerVolumeSpecName "kube-api-access-2d9p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:51:28.037242 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.037176 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca" pod "b3536bf1-150e-463b-a4f5-a5d3eff8fe63" (UID: "b3536bf1-150e-463b-a4f5-a5d3eff8fe63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:51:28.136208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.136168 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:28.136208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.136201 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9p7\" (UniqueName: \"kubernetes.io/projected/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-kube-api-access-2d9p7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:28.136208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.136212 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3536bf1-150e-463b-a4f5-a5d3eff8fe63-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:28.772374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.772345 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" Nov 03 17:51:28.772374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.772359 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5" event={"ID":"b3536bf1-150e-463b-a4f5-a5d3eff8fe63","Type":"ContainerDied","Data":"94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4"} Nov 03 17:51:28.772721 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:28.772409 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94d6eb075ed837bec69ea3e47fb439ea022559e374a67806bd16ea9d482616a4" Nov 03 17:51:52.250623 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:51:52.250578 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:51:55.014787 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.014748 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2"] Nov 03 17:51:55.016474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.016334 6001 cpu_manager.go:414] "containerMap: removing container" podUID="b3536bf1-150e-463b-a4f5-a5d3eff8fe63" containerName="revoke-keys" Nov 03 17:51:55.016474 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.016363 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3536bf1-150e-463b-a4f5-a5d3eff8fe63" containerName="revoke-keys" Nov 03 17:51:55.016697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.016587 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3536bf1-150e-463b-a4f5-a5d3eff8fe63" containerName="revoke-keys" Nov 03 17:51:55.024367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.024323 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.037538 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.037508 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2"] Nov 03 17:51:55.048026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.047979 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:51:55.048026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.047979 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:51:55.048291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.048074 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:51:55.054973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.054937 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:51:55.136970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.136931 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.137201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.136973 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.137201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.137161 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2n2n\" (UniqueName: \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.238671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.238627 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.238671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.238671 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.238973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.238752 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2n2n\" (UniqueName: \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.239567 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.239545 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.241151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.241127 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.262854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.262828 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2n2n\" (UniqueName: \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\") pod \"maas-api-revoke-sub-e2fc9c5b-pm4n2\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.339794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.339715 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:55.642931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.642900 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2"] Nov 03 17:51:55.645248 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:51:55.645215 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff0d2c8_48f7_45fe_ad19_30358fe5f582.slice/crio-4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d WatchSource:0}: Error finding container 4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d: Status 404 returned error can't find the container with id 4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d Nov 03 17:51:55.943045 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:55.942964 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" event={"ID":"2ff0d2c8-48f7-45fe-ad19-30358fe5f582","Type":"ContainerStarted","Data":"4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d"} Nov 03 17:51:56.950778 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:56.950684 6001 generic.go:356] "Generic (PLEG): container finished" podID="2ff0d2c8-48f7-45fe-ad19-30358fe5f582" containerID="41fc9a21cab69175b04376dca1e173aec4f9a3abad87fde2b7cff361aad9ad67" exitCode=0 Nov 03 17:51:56.951179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:56.950767 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" event={"ID":"2ff0d2c8-48f7-45fe-ad19-30358fe5f582","Type":"ContainerDied","Data":"41fc9a21cab69175b04376dca1e173aec4f9a3abad87fde2b7cff361aad9ad67"} Nov 03 17:51:58.095492 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.095466 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:58.268622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.268568 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\") pod \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " Nov 03 17:51:58.268831 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.268749 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\") pod \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " Nov 03 17:51:58.268902 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.268866 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\" (UniqueName: \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\") pod \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\" (UID: \"2ff0d2c8-48f7-45fe-ad19-30358fe5f582\") " Nov 03 17:51:58.269025 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.269003 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca" pod "2ff0d2c8-48f7-45fe-ad19-30358fe5f582" (UID: "2ff0d2c8-48f7-45fe-ad19-30358fe5f582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:51:58.269192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.269176 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:58.270881 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.270861 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca" pod "2ff0d2c8-48f7-45fe-ad19-30358fe5f582" (UID: "2ff0d2c8-48f7-45fe-ad19-30358fe5f582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:51:58.271019 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.270995 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n" pod "2ff0d2c8-48f7-45fe-ad19-30358fe5f582" (UID: "2ff0d2c8-48f7-45fe-ad19-30358fe5f582"). InnerVolumeSpecName "kube-api-access-p2n2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:51:58.369825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.369787 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:58.369825 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.369821 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2n2n\" (UniqueName: \"kubernetes.io/projected/2ff0d2c8-48f7-45fe-ad19-30358fe5f582-kube-api-access-p2n2n\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:51:58.965250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.965220 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" Nov 03 17:51:58.965457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.965220 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2" event={"ID":"2ff0d2c8-48f7-45fe-ad19-30358fe5f582","Type":"ContainerDied","Data":"4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d"} Nov 03 17:51:58.965457 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:51:58.965348 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb2f6fd113e09de96a5635de70168c14fe3fee8f5234ab56ad26a694328d66d" Nov 03 17:52:05.346427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.346383 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2"] Nov 03 17:52:05.347104 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.347080 6001 cpu_manager.go:414] "containerMap: removing container" podUID="2ff0d2c8-48f7-45fe-ad19-30358fe5f582" containerName="revoke-keys" Nov 03 17:52:05.347185 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.347106 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff0d2c8-48f7-45fe-ad19-30358fe5f582" containerName="revoke-keys" Nov 03 17:52:05.347247 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.347237 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff0d2c8-48f7-45fe-ad19-30358fe5f582" containerName="revoke-keys" Nov 03 17:52:05.351273 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.351249 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.370527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.370491 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2"] Nov 03 17:52:05.379736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.379703 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:52:05.379947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.379763 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:52:05.437819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.437790 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:52:05.437819 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.437790 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:52:05.442968 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.442941 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.443098 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.443000 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8jk\" (UniqueName: \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.443146 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.443134 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.544032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.543986 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.544257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.544075 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.544257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.544120 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8jk\" (UniqueName: \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.544794 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.544773 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.546427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.546392 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.568179 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.568140 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8jk\" (UniqueName: \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\") pod \"maas-api-revoke-sub-5a29efb3-wtsf2\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.665626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.665537 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:05.971165 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:05.971138 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2"] Nov 03 17:52:05.971836 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:52:05.971801 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85430cc0_adbc_4e32_8182_ee4c60984317.slice/crio-f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831 WatchSource:0}: Error finding container f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831: Status 404 returned error can't find the container with id f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831 Nov 03 17:52:06.005250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:06.005211 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" event={"ID":"85430cc0-adbc-4e32-8182-ee4c60984317","Type":"ContainerStarted","Data":"f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831"} Nov 03 17:52:07.014056 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:07.013952 6001 generic.go:356] "Generic (PLEG): container finished" podID="85430cc0-adbc-4e32-8182-ee4c60984317" containerID="5d50d4542ac3ce759741db3673aa856b0fd25a22aa4fe24500dc1340a8b0478e" exitCode=0 Nov 03 17:52:07.014493 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:07.014056 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" event={"ID":"85430cc0-adbc-4e32-8182-ee4c60984317","Type":"ContainerDied","Data":"5d50d4542ac3ce759741db3673aa856b0fd25a22aa4fe24500dc1340a8b0478e"} Nov 03 17:52:08.397207 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.396797 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:08.509469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.509392 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\") pod \"85430cc0-adbc-4e32-8182-ee4c60984317\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " Nov 03 17:52:08.509722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.509684 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\" (UniqueName: \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\") pod \"85430cc0-adbc-4e32-8182-ee4c60984317\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " Nov 03 17:52:08.509795 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.509752 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\") pod \"85430cc0-adbc-4e32-8182-ee4c60984317\" (UID: \"85430cc0-adbc-4e32-8182-ee4c60984317\") " Nov 03 17:52:08.517089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.517047 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca" pod "85430cc0-adbc-4e32-8182-ee4c60984317" (UID: "85430cc0-adbc-4e32-8182-ee4c60984317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:08.521170 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.521129 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk" pod "85430cc0-adbc-4e32-8182-ee4c60984317" (UID: "85430cc0-adbc-4e32-8182-ee4c60984317"). InnerVolumeSpecName "kube-api-access-kf8jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:08.529303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.529251 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca" pod "85430cc0-adbc-4e32-8182-ee4c60984317" (UID: "85430cc0-adbc-4e32-8182-ee4c60984317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:08.611434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.611264 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8jk\" (UniqueName: \"kubernetes.io/projected/85430cc0-adbc-4e32-8182-ee4c60984317-kube-api-access-kf8jk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:08.611434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.611321 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:08.611434 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:08.611337 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85430cc0-adbc-4e32-8182-ee4c60984317-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:09.031118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:09.031081 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" event={"ID":"85430cc0-adbc-4e32-8182-ee4c60984317","Type":"ContainerDied","Data":"f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831"} Nov 03 17:52:09.031118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:09.031114 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2" Nov 03 17:52:09.031369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:09.031133 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f82ea1eed6c2dd8022e3c132e8aa969364cf6a96a007832ecb1ba737a0fc3831" Nov 03 17:52:14.833777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:14.833742 6001 scope.go:122] "RemoveContainer" containerID="b1d926ce73e1d11c1b1f32ad68256447a8c7a4e529671511f0ed291dec2ae346" Nov 03 17:52:14.843471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:14.843432 6001 scope.go:122] "RemoveContainer" containerID="46c09d3ee5dfcc5b9a49cdd9278e61625483c33daf75b1f472ae5ad5a5d14c8c" Nov 03 17:52:22.002549 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.002511 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5"] Nov 03 17:52:22.014767 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.014739 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lwvw5"] Nov 03 17:52:22.050032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.049995 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk"] Nov 03 17:52:22.050562 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.050541 6001 cpu_manager.go:414] "containerMap: removing container" podUID="85430cc0-adbc-4e32-8182-ee4c60984317" containerName="revoke-keys" Nov 03 17:52:22.050694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.050562 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85430cc0-adbc-4e32-8182-ee4c60984317" containerName="revoke-keys" Nov 03 17:52:22.050694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.050675 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="85430cc0-adbc-4e32-8182-ee4c60984317" containerName="revoke-keys" Nov 03 17:52:22.055147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.055124 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.080338 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.080298 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk"] Nov 03 17:52:22.083511 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.083483 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:52:22.083671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.083598 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:52:22.083869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.083851 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:52:22.083981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.083961 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:52:22.159107 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.159068 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.159300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.159171 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgml5\" (UniqueName: \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.159300 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.159249 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.250091 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:52:22.250055 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:52:22.260799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.260718 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.260997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.260833 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.260997 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.260868 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgml5\" (UniqueName: \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.261576 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.261550 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.263130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.263106 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.285036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.285005 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgml5\" (UniqueName: \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\") pod \"maas-api-revoke-sub-3d4156f1-wpszk\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.368619 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.368577 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:22.667558 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.667527 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk"] Nov 03 17:52:22.667976 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:52:22.667954 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe770c2b_475f_454b_a590_9b766b1ee834.slice/crio-a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473 WatchSource:0}: Error finding container a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473: Status 404 returned error can't find the container with id a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473 Nov 03 17:52:22.788420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:22.788386 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3536bf1-150e-463b-a4f5-a5d3eff8fe63" path="/var/lib/kubelet/pods/b3536bf1-150e-463b-a4f5-a5d3eff8fe63/volumes" Nov 03 17:52:23.109534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:23.109490 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" event={"ID":"fe770c2b-475f-454b-a590-9b766b1ee834","Type":"ContainerStarted","Data":"94b922408a21b254762fc28a15070ce64fa8e9c333888269289ce87d54835913"} Nov 03 17:52:23.109962 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:23.109542 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" event={"ID":"fe770c2b-475f-454b-a590-9b766b1ee834","Type":"ContainerStarted","Data":"a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473"} Nov 03 17:52:23.151969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:23.151918 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" podStartSLOduration=1.151905421 podStartE2EDuration="1.151905421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:23.151689535 +0000 UTC m=+5838.484903105" watchObservedRunningTime="2026-11-03 17:52:23.151905421 +0000 UTC m=+5838.485118994" Nov 03 17:52:24.116011 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:24.115977 6001 generic.go:356] "Generic (PLEG): container finished" podID="fe770c2b-475f-454b-a590-9b766b1ee834" containerID="94b922408a21b254762fc28a15070ce64fa8e9c333888269289ce87d54835913" exitCode=0 Nov 03 17:52:24.116379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:24.116044 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" event={"ID":"fe770c2b-475f-454b-a590-9b766b1ee834","Type":"ContainerDied","Data":"94b922408a21b254762fc28a15070ce64fa8e9c333888269289ce87d54835913"} Nov 03 17:52:25.262861 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.262833 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:25.392885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.392784 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\") pod \"fe770c2b-475f-454b-a590-9b766b1ee834\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " Nov 03 17:52:25.392885 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.392838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\" (UniqueName: \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\") pod \"fe770c2b-475f-454b-a590-9b766b1ee834\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " Nov 03 17:52:25.393156 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.392890 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\") pod \"fe770c2b-475f-454b-a590-9b766b1ee834\" (UID: \"fe770c2b-475f-454b-a590-9b766b1ee834\") " Nov 03 17:52:25.393368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.393344 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca" pod "fe770c2b-475f-454b-a590-9b766b1ee834" (UID: "fe770c2b-475f-454b-a590-9b766b1ee834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:52:25.395119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.395095 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5" pod "fe770c2b-475f-454b-a590-9b766b1ee834" (UID: "fe770c2b-475f-454b-a590-9b766b1ee834"). InnerVolumeSpecName "kube-api-access-qgml5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:52:25.395119 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.395112 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca" pod "fe770c2b-475f-454b-a590-9b766b1ee834" (UID: "fe770c2b-475f-454b-a590-9b766b1ee834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:52:25.493789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.493755 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:25.493789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.493784 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgml5\" (UniqueName: \"kubernetes.io/projected/fe770c2b-475f-454b-a590-9b766b1ee834-kube-api-access-qgml5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:25.493789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:25.493793 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe770c2b-475f-454b-a590-9b766b1ee834-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:52:26.127925 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:26.127896 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" Nov 03 17:52:26.128149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:26.127888 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk" event={"ID":"fe770c2b-475f-454b-a590-9b766b1ee834","Type":"ContainerDied","Data":"a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473"} Nov 03 17:52:26.128149 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:26.128043 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0fe0b89a6467de1a4ae7e8403395b8e0d8fce00e91291e1f481227e59075473" Nov 03 17:52:37.540732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.540690 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:52:37.541261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.541195 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fe770c2b-475f-454b-a590-9b766b1ee834" containerName="revoke-keys" Nov 03 17:52:37.541261 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.541211 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe770c2b-475f-454b-a590-9b766b1ee834" containerName="revoke-keys" Nov 03 17:52:37.541382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.541297 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe770c2b-475f-454b-a590-9b766b1ee834" containerName="revoke-keys" Nov 03 17:52:37.544699 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.544674 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.561388 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.561349 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-27d3f0-openshift-default-dockercfg-tbtv5\"" Nov 03 17:52:37.577022 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.576983 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:52:37.740784 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.740740 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.740995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.740794 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.740995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.740881 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.740993 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.741028 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.741047 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.741069 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.741120 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krffw\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.741348 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.741151 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842302 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842364 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842415 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842387 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842424 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842487 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842508 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842532 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842565 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krffw\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.842847 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842590 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.843210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.842985 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.843210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.843038 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.843369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.843216 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.843369 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.843249 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.843513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.843418 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.845057 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.845036 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.845294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.845270 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.895854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.895812 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:37.899383 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:37.899353 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krffw\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\") pod \"e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:38.158552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:38.158423 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:38.477529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:38.477499 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:52:38.477677 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:52:38.477645 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ba018c_3d13_441d_a8dd_c74411396a29.slice/crio-8d5fdbf28e4675944b78b26eaeb045f1edb1e6b5ccc56342eb377debcc359865 WatchSource:0}: Error finding container 8d5fdbf28e4675944b78b26eaeb045f1edb1e6b5ccc56342eb377debcc359865: Status 404 returned error can't find the container with id 8d5fdbf28e4675944b78b26eaeb045f1edb1e6b5ccc56342eb377debcc359865 Nov 03 17:52:38.480005 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:38.479974 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:38.480105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:38.480039 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:38.480105 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:38.480070 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:39.231514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:39.231475 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerStarted","Data":"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6"} Nov 03 17:52:39.231951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:39.231539 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerStarted","Data":"8d5fdbf28e4675944b78b26eaeb045f1edb1e6b5ccc56342eb377debcc359865"} Nov 03 17:52:39.293152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:39.293094 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podStartSLOduration=2.293078177 podStartE2EDuration="2.293078177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:39.29074638 +0000 UTC m=+5854.623959950" watchObservedRunningTime="2026-11-03 17:52:39.293078177 +0000 UTC m=+5854.626291745" Nov 03 17:52:40.158876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:40.158840 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:40.164683 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:40.164653 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:40.238402 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:40.238366 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:40.239582 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:40.239556 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:43.605169 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.605126 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:52:43.609379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.609354 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.612514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.612487 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:52:43.616598 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.616577 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.627614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.627588 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cfc34b81-dockercfg-ghrgj\"" Nov 03 17:52:43.630333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.630309 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:52:43.639839 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.639815 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-27d3f0\"" Nov 03 17:52:43.642293 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.642272 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:52:43.693254 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.693228 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.693375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.693291 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.693426 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.693402 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpm85\" (UniqueName: \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.693482 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.693466 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjszz\" (UniqueName: \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.794845 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.794807 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpm85\" (UniqueName: \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.795075 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.794876 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjszz\" (UniqueName: \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.795205 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.795179 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.795543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.795513 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.796227 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.796201 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.796323 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.796251 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.824936 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.824907 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjszz\" (UniqueName: \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:43.825068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.824951 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpm85\" (UniqueName: \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.923979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.923876 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:43.933210 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:43.933180 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:44.241034 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:52:44.241002 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635a3e0f_6eb8_430f_8627_6264d020026b.slice/crio-53add06ed60bd7c8983e03ba63c8616df0933593554e864d733dbb7e0320221b WatchSource:0}: Error finding container 53add06ed60bd7c8983e03ba63c8616df0933593554e864d733dbb7e0320221b: Status 404 returned error can't find the container with id 53add06ed60bd7c8983e03ba63c8616df0933593554e864d733dbb7e0320221b Nov 03 17:52:44.242299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:44.242267 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:52:44.254409 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:44.254381 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:52:44.254749 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:52:44.254726 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37f9cb8_8128_4530_aff3_3ca03d503fa0.slice/crio-7c776e063f6270dfea355ba3fe6959f507a7a9ba724b8c1af8a6be3d24f1389f WatchSource:0}: Error finding container 7c776e063f6270dfea355ba3fe6959f507a7a9ba724b8c1af8a6be3d24f1389f: Status 404 returned error can't find the container with id 7c776e063f6270dfea355ba3fe6959f507a7a9ba724b8c1af8a6be3d24f1389f Nov 03 17:52:44.262877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:44.262834 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" event={"ID":"d37f9cb8-8128-4530-aff3-3ca03d503fa0","Type":"ContainerStarted","Data":"7c776e063f6270dfea355ba3fe6959f507a7a9ba724b8c1af8a6be3d24f1389f"} Nov 03 17:52:44.264203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:44.264166 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" event={"ID":"635a3e0f-6eb8-430f-8627-6264d020026b","Type":"ContainerStarted","Data":"53add06ed60bd7c8983e03ba63c8616df0933593554e864d733dbb7e0320221b"} Nov 03 17:52:45.272374 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.272331 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" event={"ID":"635a3e0f-6eb8-430f-8627-6264d020026b","Type":"ContainerStarted","Data":"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260"} Nov 03 17:52:45.272876 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.272404 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:45.274031 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.274003 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" event={"ID":"d37f9cb8-8128-4530-aff3-3ca03d503fa0","Type":"ContainerStarted","Data":"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c"} Nov 03 17:52:45.274147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.274053 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:45.315393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.315333 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" podStartSLOduration=2.31531571 podStartE2EDuration="2.31531571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:45.314974879 +0000 UTC m=+5860.648188449" watchObservedRunningTime="2026-11-03 17:52:45.31531571 +0000 UTC m=+5860.648529280" Nov 03 17:52:45.357838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:45.357787 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" podStartSLOduration=2.357774144 podStartE2EDuration="2.357774144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:45.356906326 +0000 UTC m=+5860.690119898" watchObservedRunningTime="2026-11-03 17:52:45.357774144 +0000 UTC m=+5860.690987713" Nov 03 17:52:52.276490 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:52:52.276396 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:52:56.280628 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:56.280570 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:52:56.281101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:56.280652 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:52:56.283100 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:56.283079 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:52:56.283232 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:56.283140 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:52:57.349601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.349569 6001 generic.go:356] "Generic (PLEG): container finished" podID="47ba018c-3d13-441d-a8dd-c74411396a29" containerID="fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6" exitCode=0 Nov 03 17:52:57.350091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.349628 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerDied","Data":"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6"} Nov 03 17:52:57.350091 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.350016 6001 scope.go:122] "RemoveContainer" containerID="fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6" Nov 03 17:52:57.351263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.351229 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:57.351353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.351317 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:57.351393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:57.351353 6001 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 17:52:58.374036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:58.373995 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerStarted","Data":"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef"} Nov 03 17:52:59.165471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:59.161863 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:52:59.165471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:59.162003 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" start-of-body= Nov 03 17:52:59.165471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:52:59.162060 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" Nov 03 17:53:00.163469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:00.162918 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" start-of-body= Nov 03 17:53:00.163469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:00.163004 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" Nov 03 17:53:01.165470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:01.159920 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" start-of-body= Nov 03 17:53:01.165470 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:01.159990 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" Nov 03 17:53:02.166060 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:02.166022 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" start-of-body= Nov 03 17:53:02.167151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:02.166719 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" Nov 03 17:53:03.161471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:03.160332 6001 patch_prober.go:29] interesting pod/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" start-of-body= Nov 03 17:53:03.161471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:03.160395 6001 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.210:15021/healthz/ready\": dial tcp 10.134.1.210:15021: connect: connection refused" Nov 03 17:53:04.163238 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:04.163205 6001 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:53:04.163738 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:04.163558 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:53:04.164189 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:04.164172 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:53:18.034366 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:18.034325 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj"] Nov 03 17:53:18.046483 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:18.046432 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5m6vj"] Nov 03 17:53:18.788362 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:18.788325 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a932eec1-e0be-406b-bd63-38e45a935373" path="/var/lib/kubelet/pods/a932eec1-e0be-406b-bd63-38e45a935373/volumes" Nov 03 17:53:22.251048 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:53:22.251002 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:53:30.378297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.378253 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw"] Nov 03 17:53:30.381755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.381733 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.400002 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.399960 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:53:30.400529 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.400500 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:53:30.402783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.402756 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw"] Nov 03 17:53:30.412499 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.412426 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:53:30.412707 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.412569 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:53:30.417353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.417313 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spp9s\" (UniqueName: \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.417733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.417457 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.417733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.417495 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.518883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.518826 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.518883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.518888 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.519147 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.518952 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spp9s\" (UniqueName: \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.519696 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.519664 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.521398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.521372 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.543835 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.543798 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spp9s\" (UniqueName: \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\") pod \"maas-api-revoke-sub-c1002a66-9cvvw\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:30.695297 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:30.695194 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:31.000249 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:31.000212 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw"] Nov 03 17:53:31.000724 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:53:31.000697 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8645435_031d_41cf_a879_f25ff0d2ace7.slice/crio-20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc WatchSource:0}: Error finding container 20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc: Status 404 returned error can't find the container with id 20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc Nov 03 17:53:31.583579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:31.583486 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" event={"ID":"f8645435-031d-41cf-a879-f25ff0d2ace7","Type":"ContainerStarted","Data":"f78a130d3ad54405ede2daaffab47706dd84e57ab3660fac0bc6500f55024aa6"} Nov 03 17:53:31.583579 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:31.583550 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" event={"ID":"f8645435-031d-41cf-a879-f25ff0d2ace7","Type":"ContainerStarted","Data":"20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc"} Nov 03 17:53:31.625490 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:31.625421 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" podStartSLOduration=1.625409095 podStartE2EDuration="1.625409095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:31.625220151 +0000 UTC m=+5906.958433721" watchObservedRunningTime="2026-11-03 17:53:31.625409095 +0000 UTC m=+5906.958622664" Nov 03 17:53:32.590756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:32.590717 6001 generic.go:356] "Generic (PLEG): container finished" podID="f8645435-031d-41cf-a879-f25ff0d2ace7" containerID="f78a130d3ad54405ede2daaffab47706dd84e57ab3660fac0bc6500f55024aa6" exitCode=0 Nov 03 17:53:32.591225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:32.590804 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" event={"ID":"f8645435-031d-41cf-a879-f25ff0d2ace7","Type":"ContainerDied","Data":"f78a130d3ad54405ede2daaffab47706dd84e57ab3660fac0bc6500f55024aa6"} Nov 03 17:53:33.742397 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.742366 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:33.851069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.850973 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\") pod \"f8645435-031d-41cf-a879-f25ff0d2ace7\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " Nov 03 17:53:33.851256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.851147 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\") pod \"f8645435-031d-41cf-a879-f25ff0d2ace7\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " Nov 03 17:53:33.851256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.851194 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\" (UniqueName: \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\") pod \"f8645435-031d-41cf-a879-f25ff0d2ace7\" (UID: \"f8645435-031d-41cf-a879-f25ff0d2ace7\") " Nov 03 17:53:33.851385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.851341 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca" pod "f8645435-031d-41cf-a879-f25ff0d2ace7" (UID: "f8645435-031d-41cf-a879-f25ff0d2ace7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:33.851610 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.851576 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:53:33.853406 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.853384 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s" pod "f8645435-031d-41cf-a879-f25ff0d2ace7" (UID: "f8645435-031d-41cf-a879-f25ff0d2ace7"). InnerVolumeSpecName "kube-api-access-spp9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:33.853512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.853475 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca" pod "f8645435-031d-41cf-a879-f25ff0d2ace7" (UID: "f8645435-031d-41cf-a879-f25ff0d2ace7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:33.953548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.953504 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8645435-031d-41cf-a879-f25ff0d2ace7-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:53:33.953548 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:33.953543 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spp9s\" (UniqueName: \"kubernetes.io/projected/f8645435-031d-41cf-a879-f25ff0d2ace7-kube-api-access-spp9s\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:53:34.604142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:34.604108 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" Nov 03 17:53:34.604367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:34.604141 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw" event={"ID":"f8645435-031d-41cf-a879-f25ff0d2ace7","Type":"ContainerDied","Data":"20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc"} Nov 03 17:53:34.604367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:34.604202 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20d27ec66bbe7b06232111e834e542bb239c121ac10ea5db62a64b1d4338b8dc" Nov 03 17:53:41.036240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:41.036204 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6"] Nov 03 17:53:41.048656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:41.048623 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-q4gw6"] Nov 03 17:53:42.788379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:42.788339 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3a650b-034a-494a-84f2-d91c5b6acb86" path="/var/lib/kubelet/pods/2c3a650b-034a-494a-84f2-d91c5b6acb86/volumes" Nov 03 17:53:48.449594 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:48.449553 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:53:48.480084 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:48.480050 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:53:52.252669 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:53:52.252628 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:53:53.450743 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:53.450666 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" containerID="cri-o://cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260" gracePeriod=25 Nov 03 17:53:53.480665 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:53.480621 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" containerID="cri-o://294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c" gracePeriod=25 Nov 03 17:53:56.282173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:56.282130 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:53:56.282660 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:53:56.282130 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:54:01.029914 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:01.029876 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d"] Nov 03 17:54:01.042181 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:01.042147 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-8qj2d"] Nov 03 17:54:02.788760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:02.788726 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ca48a0-5031-480e-964a-1324ad2d7b27" path="/var/lib/kubelet/pods/97ca48a0-5031-480e-964a-1324ad2d7b27/volumes" Nov 03 17:54:06.282416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:06.282377 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:54:06.282870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:06.282414 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:54:12.029039 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:12.028993 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg"] Nov 03 17:54:12.041349 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:12.041319 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-l4qdg"] Nov 03 17:54:12.788248 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:12.788213 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ba9f39-f4dc-451b-adff-78a65826dd28" path="/var/lib/kubelet/pods/66ba9f39-f4dc-451b-adff-78a65826dd28/volumes" Nov 03 17:54:13.729433 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.729410 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:54:13.757418 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.757396 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:54:13.796354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.796326 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\" (UniqueName: \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\") pod \"635a3e0f-6eb8-430f-8627-6264d020026b\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " Nov 03 17:54:13.796544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.796469 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\") pod \"635a3e0f-6eb8-430f-8627-6264d020026b\" (UID: \"635a3e0f-6eb8-430f-8627-6264d020026b\") " Nov 03 17:54:13.796878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.796856 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume" pod "635a3e0f-6eb8-430f-8627-6264d020026b" (UID: "635a3e0f-6eb8-430f-8627-6264d020026b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:13.798588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.798562 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85" pod "635a3e0f-6eb8-430f-8627-6264d020026b" (UID: "635a3e0f-6eb8-430f-8627-6264d020026b"). InnerVolumeSpecName "kube-api-access-jpm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:13.841722 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.841694 6001 generic.go:356] "Generic (PLEG): container finished" podID="635a3e0f-6eb8-430f-8627-6264d020026b" containerID="cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260" exitCode=0 Nov 03 17:54:13.841866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.841765 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" Nov 03 17:54:13.841866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.841790 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" event={"ID":"635a3e0f-6eb8-430f-8627-6264d020026b","Type":"ContainerDied","Data":"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260"} Nov 03 17:54:13.841866 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.841850 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw" event={"ID":"635a3e0f-6eb8-430f-8627-6264d020026b","Type":"ContainerDied","Data":"53add06ed60bd7c8983e03ba63c8616df0933593554e864d733dbb7e0320221b"} Nov 03 17:54:13.842049 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.841881 6001 scope.go:122] "RemoveContainer" containerID="cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260" Nov 03 17:54:13.843401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.843376 6001 generic.go:356] "Generic (PLEG): container finished" podID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerID="294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c" exitCode=0 Nov 03 17:54:13.843556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.843434 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" Nov 03 17:54:13.843556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.843489 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" event={"ID":"d37f9cb8-8128-4530-aff3-3ca03d503fa0","Type":"ContainerDied","Data":"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c"} Nov 03 17:54:13.843556 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.843528 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn" event={"ID":"d37f9cb8-8128-4530-aff3-3ca03d503fa0","Type":"ContainerDied","Data":"7c776e063f6270dfea355ba3fe6959f507a7a9ba724b8c1af8a6be3d24f1389f"} Nov 03 17:54:13.852527 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.852506 6001 scope.go:122] "RemoveContainer" containerID="cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260" Nov 03 17:54:13.852851 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:13.852826 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260\": container with ID starting with cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260 not found: ID does not exist" containerID="cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260" Nov 03 17:54:13.852943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.852863 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260"} err="failed to get container status \"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260\": rpc error: code = NotFound desc = could not find container \"cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260\": container with ID starting with cbd830a6ee83c41a76044853cf324c2023cc604cad82e06371c940e9650ec260 not found: ID does not exist" Nov 03 17:54:13.852943 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.852880 6001 scope.go:122] "RemoveContainer" containerID="294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c" Nov 03 17:54:13.861280 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.861264 6001 scope.go:122] "RemoveContainer" containerID="294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c" Nov 03 17:54:13.861546 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:13.861525 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c\": container with ID starting with 294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c not found: ID does not exist" containerID="294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c" Nov 03 17:54:13.861617 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.861555 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c"} err="failed to get container status \"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c\": rpc error: code = NotFound desc = could not find container \"294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c\": container with ID starting with 294c5dfbd0180926fbd84e6ae324cccf853c334ef3537a7eb4dbaa0ea9e5221c not found: ID does not exist" Nov 03 17:54:13.897829 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.897794 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\" (UniqueName: \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\") pod \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " Nov 03 17:54:13.898036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.897838 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\") pod \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\" (UID: \"d37f9cb8-8128-4530-aff3-3ca03d503fa0\") " Nov 03 17:54:13.898115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.898043 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpm85\" (UniqueName: \"kubernetes.io/projected/635a3e0f-6eb8-430f-8627-6264d020026b-kube-api-access-jpm85\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:13.898115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.898056 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/635a3e0f-6eb8-430f-8627-6264d020026b-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:13.898327 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.898303 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume" pod "d37f9cb8-8128-4530-aff3-3ca03d503fa0" (UID: "d37f9cb8-8128-4530-aff3-3ca03d503fa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:13.899931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.899913 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz" pod "d37f9cb8-8128-4530-aff3-3ca03d503fa0" (UID: "d37f9cb8-8128-4530-aff3-3ca03d503fa0"). InnerVolumeSpecName "kube-api-access-qjszz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:13.911379 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.911347 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:54:13.915243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.915216 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f5vgtfw"] Nov 03 17:54:13.999203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.999155 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjszz\" (UniqueName: \"kubernetes.io/projected/d37f9cb8-8128-4530-aff3-3ca03d503fa0-kube-api-access-qjszz\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:13.999203 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:13.999201 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d37f9cb8-8128-4530-aff3-3ca03d503fa0-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:14.189514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:14.189468 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:54:14.193954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:14.193921 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799cx8tnn"] Nov 03 17:54:14.789163 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:14.789125 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" path="/var/lib/kubelet/pods/635a3e0f-6eb8-430f-8627-6264d020026b/volumes" Nov 03 17:54:14.789586 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:14.789545 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" path="/var/lib/kubelet/pods/d37f9cb8-8128-4530-aff3-3ca03d503fa0/volumes" Nov 03 17:54:14.879678 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:14.879647 6001 scope.go:122] "RemoveContainer" containerID="58c51bb08131414437a90ab7c0a95d7ea1ca44e8cde1e86ae128a526f49ace12" Nov 03 17:54:22.065420 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065313 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:22.065855 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065837 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" Nov 03 17:54:22.065909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065856 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" Nov 03 17:54:22.065909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065877 6001 cpu_manager.go:414] "containerMap: removing container" podUID="f8645435-031d-41cf-a879-f25ff0d2ace7" containerName="revoke-keys" Nov 03 17:54:22.065909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065882 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8645435-031d-41cf-a879-f25ff0d2ace7" containerName="revoke-keys" Nov 03 17:54:22.065909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065901 6001 cpu_manager.go:414] "containerMap: removing container" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" Nov 03 17:54:22.065909 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065908 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" Nov 03 17:54:22.066069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065975 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d37f9cb8-8128-4530-aff3-3ca03d503fa0" containerName="payload-pre-processing" Nov 03 17:54:22.066069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065987 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8645435-031d-41cf-a879-f25ff0d2ace7" containerName="revoke-keys" Nov 03 17:54:22.066069 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.065996 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="635a3e0f-6eb8-430f-8627-6264d020026b" containerName="payload-processing" Nov 03 17:54:22.070961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.070941 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.096162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.096122 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:22.099969 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.099947 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.106335 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.106305 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-27d3f0\"" Nov 03 17:54:22.106534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.106379 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cfc34b81-dockercfg-nzcjq\"" Nov 03 17:54:22.112789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.112759 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:22.126821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.126787 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:22.186533 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.186486 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.186930 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.186896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv99l\" (UniqueName: \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.252144 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:22.252096 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:54:22.288063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.288020 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv99l\" (UniqueName: \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.288285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.288075 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.288285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.288101 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.288285 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.288161 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds2h\" (UniqueName: \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.288913 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.288891 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.311827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.311788 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv99l\" (UniqueName: \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\") pod \"payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.384419 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.384323 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.388871 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.388844 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.388979 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.388963 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds2h\" (UniqueName: \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.389566 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.389549 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.412328 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.412297 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds2h\" (UniqueName: \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\") pod \"payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.417157 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.417121 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.690344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.690291 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:22.691679 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:54:22.691647 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894955db_61fe_4f1d_87cd_f82fbd8d2ad0.slice/crio-4120fe1e4e542b1ac613accfa4324b7c022769799e14993301052b6fea8c8ceb WatchSource:0}: Error finding container 4120fe1e4e542b1ac613accfa4324b7c022769799e14993301052b6fea8c8ceb: Status 404 returned error can't find the container with id 4120fe1e4e542b1ac613accfa4324b7c022769799e14993301052b6fea8c8ceb Nov 03 17:54:22.737068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.737042 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:22.741495 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:54:22.741468 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b83460f_1017_4ce2_a2a3_35c67db98b88.slice/crio-70e3d271494e8b5d19501373c1763cb2d9811007bb04fde5539076a29b8dcd96 WatchSource:0}: Error finding container 70e3d271494e8b5d19501373c1763cb2d9811007bb04fde5539076a29b8dcd96: Status 404 returned error can't find the container with id 70e3d271494e8b5d19501373c1763cb2d9811007bb04fde5539076a29b8dcd96 Nov 03 17:54:22.898216 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.898172 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" event={"ID":"894955db-61fe-4f1d-87cd-f82fbd8d2ad0","Type":"ContainerStarted","Data":"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322"} Nov 03 17:54:22.898430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.898244 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:22.898430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.898266 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" event={"ID":"894955db-61fe-4f1d-87cd-f82fbd8d2ad0","Type":"ContainerStarted","Data":"4120fe1e4e542b1ac613accfa4324b7c022769799e14993301052b6fea8c8ceb"} Nov 03 17:54:22.899928 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.899897 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" event={"ID":"5b83460f-1017-4ce2-a2a3-35c67db98b88","Type":"ContainerStarted","Data":"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6"} Nov 03 17:54:22.900072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.899950 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" event={"ID":"5b83460f-1017-4ce2-a2a3-35c67db98b88","Type":"ContainerStarted","Data":"70e3d271494e8b5d19501373c1763cb2d9811007bb04fde5539076a29b8dcd96"} Nov 03 17:54:22.900072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.899973 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:22.990756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.990695 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" podStartSLOduration=1.990678733 podStartE2EDuration="1.990678733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:22.946314041 +0000 UTC m=+5958.279527609" watchObservedRunningTime="2026-11-03 17:54:22.990678733 +0000 UTC m=+5958.323892302" Nov 03 17:54:22.991024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:22.990992 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" podStartSLOduration=0.990982801 podStartE2EDuration="990.982801ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:22.988995254 +0000 UTC m=+5958.322208825" watchObservedRunningTime="2026-11-03 17:54:22.990982801 +0000 UTC m=+5958.324196372" Nov 03 17:54:30.275820 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.275783 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp"] Nov 03 17:54:30.280296 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.280268 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.298367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.298334 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:54:30.310782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.310747 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:54:30.310782 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.310776 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:54:30.311008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.310755 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:54:30.313047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.313026 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp"] Nov 03 17:54:30.358494 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.358426 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.358716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.358607 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.358716 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.358661 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv84k\" (UniqueName: \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.460101 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.460059 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.460320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.460122 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv84k\" (UniqueName: \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.460320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.460224 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.460781 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.460760 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.462659 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.462637 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.484128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.484097 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv84k\" (UniqueName: \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\") pod \"maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.595235 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.595136 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:30.904906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.904881 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp"] Nov 03 17:54:30.906027 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:54:30.905998 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7c0fbf_2dd9_4eaa_9730_f4c3c1564422.slice/crio-a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b WatchSource:0}: Error finding container a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b: Status 404 returned error can't find the container with id a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b Nov 03 17:54:30.955923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:30.955892 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" event={"ID":"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422","Type":"ContainerStarted","Data":"a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b"} Nov 03 17:54:31.962799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:31.962762 6001 generic.go:356] "Generic (PLEG): container finished" podID="3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" containerID="cf3ab03498d1dbcedcf04571c4d6aa9acb65bab8ab7a1952d35323b04a2c9012" exitCode=0 Nov 03 17:54:31.963208 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:31.962834 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" event={"ID":"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422","Type":"ContainerDied","Data":"cf3ab03498d1dbcedcf04571c4d6aa9acb65bab8ab7a1952d35323b04a2c9012"} Nov 03 17:54:33.102173 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.102145 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:33.188463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.188428 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\") pod \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " Nov 03 17:54:33.188671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.188504 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\") pod \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " Nov 03 17:54:33.188671 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.188550 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\" (UniqueName: \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\") pod \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\" (UID: \"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422\") " Nov 03 17:54:33.188927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.188902 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca" pod "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" (UID: "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:33.189034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.189018 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:33.190593 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.190569 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca" pod "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" (UID: "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:33.190709 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.190687 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k" pod "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" (UID: "3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422"). InnerVolumeSpecName "kube-api-access-vv84k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:33.290230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.290196 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:33.290230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.290229 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv84k\" (UniqueName: \"kubernetes.io/projected/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422-kube-api-access-vv84k\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:33.908812 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.908784 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:33.909029 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.908847 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:33.978341 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.978309 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" Nov 03 17:54:33.978521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.978308 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp" event={"ID":"3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422","Type":"ContainerDied","Data":"a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b"} Nov 03 17:54:33.978521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:33.978495 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a368ca7bbf4c18ee8ec251434fe02ba38db7065d5bb3f036f2cb0dc9077c6c8b" Nov 03 17:54:40.028219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:40.028185 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh"] Nov 03 17:54:40.040843 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:40.040809 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-mvxgh"] Nov 03 17:54:40.702440 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:40.702405 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:40.734000 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:40.733972 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:40.793093 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:40.793054 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c56f89-5e8b-4ef5-a527-1dab9878db1b" path="/var/lib/kubelet/pods/40c56f89-5e8b-4ef5-a527-1dab9878db1b/volumes" Nov 03 17:54:43.023989 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:43.023952 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp"] Nov 03 17:54:43.036411 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:43.036374 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-27d3f0-vgqqp"] Nov 03 17:54:43.307513 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:43.307410 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:54:43.308403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:43.308366 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" containerID="cri-o://d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef" gracePeriod=30 Nov 03 17:54:44.525868 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.525832 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk"] Nov 03 17:54:44.538314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.538284 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wpszk"] Nov 03 17:54:44.580471 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.580421 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4"] Nov 03 17:54:44.580945 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.580932 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" containerName="revoke-keys" Nov 03 17:54:44.580995 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.580946 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" containerName="revoke-keys" Nov 03 17:54:44.581041 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.581030 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" containerName="revoke-keys" Nov 03 17:54:44.584606 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.584585 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.602505 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.602471 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:54:44.603125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.602601 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:54:44.603125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.602668 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:54:44.603125 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.602711 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:54:44.605233 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.605208 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4"] Nov 03 17:54:44.726770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.726730 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.726963 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.726800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.727008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.726948 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.789013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.788915 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422" path="/var/lib/kubelet/pods/3a7c0fbf-2dd9-4eaa-9730-f4c3c1564422/volumes" Nov 03 17:54:44.789404 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.789387 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe770c2b-475f-454b-a590-9b766b1ee834" path="/var/lib/kubelet/pods/fe770c2b-475f-454b-a590-9b766b1ee834/volumes" Nov 03 17:54:44.828218 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.828183 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.828408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.828332 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.828408 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.828384 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.829065 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.829028 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.830688 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.830668 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.851965 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.851937 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\") pod \"maas-api-revoke-sub-3d4156f1-27pw4\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:44.897981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:44.897940 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:45.205036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:45.205001 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4"] Nov 03 17:54:45.206459 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:54:45.206416 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0eb223b_120a_420e_898a_b3dbaedd2cc0.slice/crio-9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4 WatchSource:0}: Error finding container 9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4: Status 404 returned error can't find the container with id 9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4 Nov 03 17:54:45.703599 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:45.703552 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" podUID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" containerName="payload-processing" containerID="cri-o://f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322" gracePeriod=25 Nov 03 17:54:45.734872 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:45.734809 6001 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" podUID="5b83460f-1017-4ce2-a2a3-35c67db98b88" containerName="payload-pre-processing" containerID="cri-o://c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6" gracePeriod=25 Nov 03 17:54:45.998799 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:45.998770 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:46.028648 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.028619 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:46.051878 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.051823 6001 generic.go:356] "Generic (PLEG): container finished" podID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" containerID="f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322" exitCode=0 Nov 03 17:54:46.052077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.051880 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" event={"ID":"894955db-61fe-4f1d-87cd-f82fbd8d2ad0","Type":"ContainerDied","Data":"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322"} Nov 03 17:54:46.052077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.051946 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" event={"ID":"894955db-61fe-4f1d-87cd-f82fbd8d2ad0","Type":"ContainerDied","Data":"4120fe1e4e542b1ac613accfa4324b7c022769799e14993301052b6fea8c8ceb"} Nov 03 17:54:46.052077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.051890 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978" Nov 03 17:54:46.052077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.051978 6001 scope.go:122] "RemoveContainer" containerID="f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322" Nov 03 17:54:46.053772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.053587 6001 generic.go:356] "Generic (PLEG): container finished" podID="5b83460f-1017-4ce2-a2a3-35c67db98b88" containerID="c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6" exitCode=0 Nov 03 17:54:46.053772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.053628 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" Nov 03 17:54:46.053772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.053640 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" event={"ID":"5b83460f-1017-4ce2-a2a3-35c67db98b88","Type":"ContainerDied","Data":"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6"} Nov 03 17:54:46.053772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.053717 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw" event={"ID":"5b83460f-1017-4ce2-a2a3-35c67db98b88","Type":"ContainerDied","Data":"70e3d271494e8b5d19501373c1763cb2d9811007bb04fde5539076a29b8dcd96"} Nov 03 17:54:46.055501 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.055475 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" event={"ID":"e0eb223b-120a-420e-898a-b3dbaedd2cc0","Type":"ContainerStarted","Data":"2979c7fbc015e47ae919839ed7d6db610ea6815bcf8585e81aca42d1d3e7ecac"} Nov 03 17:54:46.055625 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.055529 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" event={"ID":"e0eb223b-120a-420e-898a-b3dbaedd2cc0","Type":"ContainerStarted","Data":"9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4"} Nov 03 17:54:46.062270 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.062242 6001 scope.go:122] "RemoveContainer" containerID="f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322" Nov 03 17:54:46.062591 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:46.062566 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322\": container with ID starting with f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322 not found: ID does not exist" containerID="f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322" Nov 03 17:54:46.064050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.062600 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322"} err="failed to get container status \"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322\": rpc error: code = NotFound desc = could not find container \"f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322\": container with ID starting with f0fd35935fe4830af56ddf092c9b73b5ece0faa149b30baf1e56628766477322 not found: ID does not exist" Nov 03 17:54:46.064050 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.062624 6001 scope.go:122] "RemoveContainer" containerID="c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6" Nov 03 17:54:46.073864 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.073837 6001 scope.go:122] "RemoveContainer" containerID="c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6" Nov 03 17:54:46.074278 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:46.074248 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6\": container with ID starting with c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6 not found: ID does not exist" containerID="c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6" Nov 03 17:54:46.074387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.074294 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6"} err="failed to get container status \"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6\": rpc error: code = NotFound desc = could not find container \"c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6\": container with ID starting with c5d1028639503a6c3f5f28e339697bf73cc9254cd978393bf9d36c5359e12cb6 not found: ID does not exist" Nov 03 17:54:46.128086 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.128028 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" podStartSLOduration=2.12801208 podStartE2EDuration="2.12801208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:46.126839167 +0000 UTC m=+5981.460052737" watchObservedRunningTime="2026-11-03 17:54:46.12801208 +0000 UTC m=+5981.461225645" Nov 03 17:54:46.143957 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.143927 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\") pod \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " Nov 03 17:54:46.144121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144006 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\" (UniqueName: \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\") pod \"5b83460f-1017-4ce2-a2a3-35c67db98b88\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " Nov 03 17:54:46.144121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144033 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\" (UniqueName: \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\") pod \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\" (UID: \"894955db-61fe-4f1d-87cd-f82fbd8d2ad0\") " Nov 03 17:54:46.144121 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144086 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\") pod \"5b83460f-1017-4ce2-a2a3-35c67db98b88\" (UID: \"5b83460f-1017-4ce2-a2a3-35c67db98b88\") " Nov 03 17:54:46.144640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144502 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume" pod "894955db-61fe-4f1d-87cd-f82fbd8d2ad0" (UID: "894955db-61fe-4f1d-87cd-f82fbd8d2ad0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:46.144640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144599 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:46.144640 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.144602 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume" pod "5b83460f-1017-4ce2-a2a3-35c67db98b88" (UID: "5b83460f-1017-4ce2-a2a3-35c67db98b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:46.146381 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.146358 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h" pod "5b83460f-1017-4ce2-a2a3-35c67db98b88" (UID: "5b83460f-1017-4ce2-a2a3-35c67db98b88"). InnerVolumeSpecName "kube-api-access-mds2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:46.146518 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.146382 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l" pod "894955db-61fe-4f1d-87cd-f82fbd8d2ad0" (UID: "894955db-61fe-4f1d-87cd-f82fbd8d2ad0"). InnerVolumeSpecName "kube-api-access-qv99l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:46.245929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.245896 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mds2h\" (UniqueName: \"kubernetes.io/projected/5b83460f-1017-4ce2-a2a3-35c67db98b88-kube-api-access-mds2h\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:46.245929 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.245926 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv99l\" (UniqueName: \"kubernetes.io/projected/894955db-61fe-4f1d-87cd-f82fbd8d2ad0-kube-api-access-qv99l\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:46.246152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.245942 6001 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b83460f-1017-4ce2-a2a3-35c67db98b88-plugins-config-volume\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:46.401020 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.400982 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:46.405326 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.405296 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-27d3f0-fc49cd5f588978"] Nov 03 17:54:46.440367 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.440339 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:46.450314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.450270 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-27d3f0-5799c4vglw"] Nov 03 17:54:46.788737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.788691 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b83460f-1017-4ce2-a2a3-35c67db98b88" path="/var/lib/kubelet/pods/5b83460f-1017-4ce2-a2a3-35c67db98b88/volumes" Nov 03 17:54:46.789301 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:46.789282 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" path="/var/lib/kubelet/pods/894955db-61fe-4f1d-87cd-f82fbd8d2ad0/volumes" Nov 03 17:54:47.065674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:47.065571 6001 generic.go:356] "Generic (PLEG): container finished" podID="e0eb223b-120a-420e-898a-b3dbaedd2cc0" containerID="2979c7fbc015e47ae919839ed7d6db610ea6815bcf8585e81aca42d1d3e7ecac" exitCode=0 Nov 03 17:54:47.065674 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:47.065631 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" event={"ID":"e0eb223b-120a-420e-898a-b3dbaedd2cc0","Type":"ContainerDied","Data":"2979c7fbc015e47ae919839ed7d6db610ea6815bcf8585e81aca42d1d3e7ecac"} Nov 03 17:54:48.216759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.216732 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:48.371817 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.371774 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\") pod \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " Nov 03 17:54:48.372015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.371856 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\") pod \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " Nov 03 17:54:48.372015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.371944 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\") pod \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\" (UID: \"e0eb223b-120a-420e-898a-b3dbaedd2cc0\") " Nov 03 17:54:48.372192 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.372169 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca" pod "e0eb223b-120a-420e-898a-b3dbaedd2cc0" (UID: "e0eb223b-120a-420e-898a-b3dbaedd2cc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:48.372340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.372318 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.374058 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.374034 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca" pod "e0eb223b-120a-420e-898a-b3dbaedd2cc0" (UID: "e0eb223b-120a-420e-898a-b3dbaedd2cc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:48.374164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.374085 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp" pod "e0eb223b-120a-420e-898a-b3dbaedd2cc0" (UID: "e0eb223b-120a-420e-898a-b3dbaedd2cc0"). InnerVolumeSpecName "kube-api-access-872rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:48.473783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.473750 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0eb223b-120a-420e-898a-b3dbaedd2cc0-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.473783 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.473780 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872rp\" (UniqueName: \"kubernetes.io/projected/e0eb223b-120a-420e-898a-b3dbaedd2cc0-kube-api-access-872rp\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.565719 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.565692 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:54:48.676329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676228 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676329 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676284 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676353 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676410 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676438 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676528 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676559 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676595 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:48.676970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676612 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676679 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:48.676970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676746 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\") pod \"47ba018c-3d13-441d-a8dd-c74411396a29\" (UID: \"47ba018c-3d13-441d-a8dd-c74411396a29\") " Nov 03 17:54:48.676970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.676853 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:48.677211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677190 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677316 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677701 6001 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-credential-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677719 6001 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-certs\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677728 6001 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-workload-socket\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677737 6001 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-data\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.677904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.677746 6001 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47ba018c-3d13-441d-a8dd-c74411396a29-istiod-ca-cert\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.678982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.678955 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:54:48.679077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.679053 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "kube-api-access-krffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:48.679252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.679232 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:48.679252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.679249 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy" pod "47ba018c-3d13-441d-a8dd-c74411396a29" (UID: "47ba018c-3d13-441d-a8dd-c74411396a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:54:48.779240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.779203 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krffw\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-kube-api-access-krffw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.779240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.779236 6001 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47ba018c-3d13-441d-a8dd-c74411396a29-istio-podinfo\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.779240 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.779247 6001 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ba018c-3d13-441d-a8dd-c74411396a29-istio-token\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:48.779531 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:48.779255 6001 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47ba018c-3d13-441d-a8dd-c74411396a29-istio-envoy\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:54:49.079613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.079581 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" Nov 03 17:54:49.079793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.079579 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4" event={"ID":"e0eb223b-120a-420e-898a-b3dbaedd2cc0","Type":"ContainerDied","Data":"9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4"} Nov 03 17:54:49.079793 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.079755 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da1b4e5bac613a6b212c1056cf569c65fb2a0d08c45b3405681365be510dcf4" Nov 03 17:54:49.081387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.081356 6001 generic.go:356] "Generic (PLEG): container finished" podID="47ba018c-3d13-441d-a8dd-c74411396a29" containerID="d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef" exitCode=0 Nov 03 17:54:49.081624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.081395 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerDied","Data":"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef"} Nov 03 17:54:49.081624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.081427 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" Nov 03 17:54:49.081624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.081473 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67" event={"ID":"47ba018c-3d13-441d-a8dd-c74411396a29","Type":"ContainerDied","Data":"8d5fdbf28e4675944b78b26eaeb045f1edb1e6b5ccc56342eb377debcc359865"} Nov 03 17:54:49.081624 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.081508 6001 scope.go:122] "RemoveContainer" containerID="d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef" Nov 03 17:54:49.091398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.091375 6001 scope.go:122] "RemoveContainer" containerID="fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6" Nov 03 17:54:49.100985 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.100960 6001 scope.go:122] "RemoveContainer" containerID="d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef" Nov 03 17:54:49.101280 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:49.101259 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef\": container with ID starting with d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef not found: ID does not exist" containerID="d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef" Nov 03 17:54:49.101360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.101293 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef"} err="failed to get container status \"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef\": rpc error: code = NotFound desc = could not find container \"d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef\": container with ID starting with d44b2f2c45a5158d958ea4100723422736046fedaa6d4ff3a21df577ac6450ef not found: ID does not exist" Nov 03 17:54:49.101360 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.101315 6001 scope.go:122] "RemoveContainer" containerID="fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6" Nov 03 17:54:49.101613 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:49.101589 6001 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6\": container with ID starting with fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6 not found: ID does not exist" containerID="fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6" Nov 03 17:54:49.101677 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.101621 6001 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6"} err="failed to get container status \"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6\": rpc error: code = NotFound desc = could not find container \"fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6\": container with ID starting with fd851c692023fd8be82cd96340821c5ff8bd2e96661368ec862d9d7b751e13c6 not found: ID does not exist" Nov 03 17:54:49.150641 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.150596 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:54:49.154287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:49.154261 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-27d3f0-openshift-default-74d69c6fd5frd67"] Nov 03 17:54:50.788882 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:50.788847 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" path="/var/lib/kubelet/pods/47ba018c-3d13-441d-a8dd-c74411396a29/volumes" Nov 03 17:54:51.029697 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:51.029657 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j"] Nov 03 17:54:51.042225 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:51.042125 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-pp47j"] Nov 03 17:54:52.251092 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:54:52.251055 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:54:52.788387 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:54:52.788351 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0170d915-8c7a-4e77-bf5c-08aa879c6a5b" path="/var/lib/kubelet/pods/0170d915-8c7a-4e77-bf5c-08aa879c6a5b/volumes" Nov 03 17:55:05.975568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:05.975542 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:55:06.095068 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:06.095042 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 17:55:12.029892 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:12.029857 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk"] Nov 03 17:55:12.042152 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:12.042123 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8dksk"] Nov 03 17:55:12.788601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:12.788566 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea23e79e-3f23-46c5-8ab5-6e226e309a4f" path="/var/lib/kubelet/pods/ea23e79e-3f23-46c5-8ab5-6e226e309a4f/volumes" Nov 03 17:55:14.913423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:14.913386 6001 scope.go:122] "RemoveContainer" containerID="f9e2eb59f505f8ab4b21c6bc646031672146f27708d785973ffa314e21d88bd5" Nov 03 17:55:14.923601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:14.923583 6001 scope.go:122] "RemoveContainer" containerID="f23f4295e38211adf57dd3432118e94840e7201900aed6000a6ba50e039f9e5f" Nov 03 17:55:14.931944 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:14.931923 6001 scope.go:122] "RemoveContainer" containerID="fe3c14b88650acd85b2c1589eee746e08761c6f651f8c1dae5d043d24fabb3fc" Nov 03 17:55:22.250090 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:55:22.250054 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:55:29.035201 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:29.035166 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59"] Nov 03 17:55:29.048130 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:29.048095 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-5nf59"] Nov 03 17:55:30.788061 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:30.788023 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f26b91-190b-4a03-af4d-f1b3a846c3e4" path="/var/lib/kubelet/pods/24f26b91-190b-4a03-af4d-f1b3a846c3e4/volumes" Nov 03 17:55:46.029949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:46.029911 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2"] Nov 03 17:55:46.042304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:46.042269 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mg8c2"] Nov 03 17:55:46.788044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:55:46.788007 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ece5404-2365-4aaa-96de-5ab03e3e1b81" path="/var/lib/kubelet/pods/9ece5404-2365-4aaa-96de-5ab03e3e1b81/volumes" Nov 03 17:55:52.251909 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:55:52.251868 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:56:00.035090 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:00.035053 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx"] Nov 03 17:56:00.046572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:00.046535 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7wpxx"] Nov 03 17:56:00.788095 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:00.788060 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3f3fd3-394b-4202-8540-ceafaf235a73" path="/var/lib/kubelet/pods/ee3f3fd3-394b-4202-8540-ceafaf235a73/volumes" Nov 03 17:56:14.975754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:14.975711 6001 scope.go:122] "RemoveContainer" containerID="6d40e1bd25e3e33934872c53d19f47f1d8f5d3b041615b11f5229d793a44b56e" Nov 03 17:56:14.984587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:14.984567 6001 scope.go:122] "RemoveContainer" containerID="a7f16bc2c963ee3456bce0a118977b5b98b085b7fa1caceed5a89d23ecbe4e79" Nov 03 17:56:14.993345 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:14.993323 6001 scope.go:122] "RemoveContainer" containerID="1c9c2c502842c2c9519f23fe0be7bcd23d23ab0be94ffcb3d2906893312951aa" Nov 03 17:56:22.250166 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:56:22.250130 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:56:29.195347 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195305 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg"] Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195762 6001 cpu_manager.go:414] "containerMap: removing container" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195776 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195792 6001 cpu_manager.go:414] "containerMap: removing container" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195797 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195806 6001 cpu_manager.go:414] "containerMap: removing container" podUID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" containerName="payload-processing" Nov 03 17:56:29.195809 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195812 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" containerName="payload-processing" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195822 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e0eb223b-120a-420e-898a-b3dbaedd2cc0" containerName="revoke-keys" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195827 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0eb223b-120a-420e-898a-b3dbaedd2cc0" containerName="revoke-keys" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195843 6001 cpu_manager.go:414] "containerMap: removing container" podUID="5b83460f-1017-4ce2-a2a3-35c67db98b88" containerName="payload-pre-processing" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195847 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b83460f-1017-4ce2-a2a3-35c67db98b88" containerName="payload-pre-processing" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195920 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195928 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ba018c-3d13-441d-a8dd-c74411396a29" containerName="istio-proxy" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195938 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="894955db-61fe-4f1d-87cd-f82fbd8d2ad0" containerName="payload-processing" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195944 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0eb223b-120a-420e-898a-b3dbaedd2cc0" containerName="revoke-keys" Nov 03 17:56:29.196034 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.195951 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b83460f-1017-4ce2-a2a3-35c67db98b88" containerName="payload-pre-processing" Nov 03 17:56:29.199262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.199240 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.214416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.214388 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:56:29.214601 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.214509 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:56:29.227044 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.227011 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:56:29.227290 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.227156 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:56:29.229048 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.229018 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg"] Nov 03 17:56:29.294815 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.294755 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.295033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.294841 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdzk\" (UniqueName: \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.295033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.294971 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.396322 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.396261 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdzk\" (UniqueName: \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.396587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.396377 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.396587 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.396487 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.397188 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.397162 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.398791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.398764 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.420541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.420514 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdzk\" (UniqueName: \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\") pod \"maas-api-revoke-sub-d576e68b-462bg\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.512771 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.512731 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:29.814865 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:29.814833 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg"] Nov 03 17:56:29.815748 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:56:29.815720 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb85b285_068e_458f_8286_77dc14b565aa.slice/crio-34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c WatchSource:0}: Error finding container 34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c: Status 404 returned error can't find the container with id 34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c Nov 03 17:56:30.668966 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:30.668930 6001 generic.go:356] "Generic (PLEG): container finished" podID="cb85b285-068e-458f-8286-77dc14b565aa" containerID="0227e1bc1977ecdd89e8c03e0fa3ec2559ad7eb5f4a9b6cfdb8b066a934f3c78" exitCode=0 Nov 03 17:56:30.669385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:30.669005 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" event={"ID":"cb85b285-068e-458f-8286-77dc14b565aa","Type":"ContainerDied","Data":"0227e1bc1977ecdd89e8c03e0fa3ec2559ad7eb5f4a9b6cfdb8b066a934f3c78"} Nov 03 17:56:30.669385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:30.669043 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" event={"ID":"cb85b285-068e-458f-8286-77dc14b565aa","Type":"ContainerStarted","Data":"34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c"} Nov 03 17:56:31.830563 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.830537 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:31.923106 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.923075 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\") pod \"cb85b285-068e-458f-8286-77dc14b565aa\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " Nov 03 17:56:31.923299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.923146 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\" (UniqueName: \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\") pod \"cb85b285-068e-458f-8286-77dc14b565aa\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " Nov 03 17:56:31.923299 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.923210 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\") pod \"cb85b285-068e-458f-8286-77dc14b565aa\" (UID: \"cb85b285-068e-458f-8286-77dc14b565aa\") " Nov 03 17:56:31.923458 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.923419 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca" pod "cb85b285-068e-458f-8286-77dc14b565aa" (UID: "cb85b285-068e-458f-8286-77dc14b565aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:56:31.923773 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.923746 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:31.925353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.925295 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk" pod "cb85b285-068e-458f-8286-77dc14b565aa" (UID: "cb85b285-068e-458f-8286-77dc14b565aa"). InnerVolumeSpecName "kube-api-access-kjdzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:56:31.925459 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:31.925363 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca" pod "cb85b285-068e-458f-8286-77dc14b565aa" (UID: "cb85b285-068e-458f-8286-77dc14b565aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:56:32.024690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:32.024643 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdzk\" (UniqueName: \"kubernetes.io/projected/cb85b285-068e-458f-8286-77dc14b565aa-kube-api-access-kjdzk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:32.024690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:32.024682 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb85b285-068e-458f-8286-77dc14b565aa-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:32.681535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:32.681500 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" Nov 03 17:56:32.681769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:32.681499 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg" event={"ID":"cb85b285-068e-458f-8286-77dc14b565aa","Type":"ContainerDied","Data":"34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c"} Nov 03 17:56:32.681769 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:32.681622 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34ece995bcb939129a20839cb6a5a19fa54556ab6a7f6ea392852692b581e59c" Nov 03 17:56:43.971252 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.971207 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79"] Nov 03 17:56:43.971910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.971886 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cb85b285-068e-458f-8286-77dc14b565aa" containerName="revoke-keys" Nov 03 17:56:43.971910 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.971910 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb85b285-068e-458f-8286-77dc14b565aa" containerName="revoke-keys" Nov 03 17:56:43.972116 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.972061 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb85b285-068e-458f-8286-77dc14b565aa" containerName="revoke-keys" Nov 03 17:56:43.976124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.976101 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:43.996922 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:43.996895 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79"] Nov 03 17:56:44.006905 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.006880 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:56:44.007584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.007562 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:56:44.007904 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.007696 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:56:44.008015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.007969 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:56:44.148695 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.148654 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.148920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.148731 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.148920 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.148840 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8ht\" (UniqueName: \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.250622 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.250568 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.250838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.250665 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8ht\" (UniqueName: \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.250838 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.250822 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.251303 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.251280 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.253948 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.253919 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.274951 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.274916 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8ht\" (UniqueName: \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\") pod \"maas-api-revoke-sub-d2bc93bc-7bc79\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.294353 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.294323 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:44.697685 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.677000 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79"] Nov 03 17:56:44.827421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:44.827381 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" event={"ID":"cd241444-9fa1-4750-9cd5-23ea00f38d71","Type":"ContainerStarted","Data":"c5091f179ab2cd67517f1139b30bf609e3428c790bbffe8a8c3fcd8c69390c40"} Nov 03 17:56:45.723412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:45.723361 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:56:45.724079 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:45.724045 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:45.831614 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:45.831577 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" event={"ID":"cd241444-9fa1-4750-9cd5-23ea00f38d71","Type":"ContainerStarted","Data":"99da88aedf01df45a1657333f2d0407ed387972b302e706de66e52a87fe671aa"} Nov 03 17:56:45.880487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:45.880405 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" podStartSLOduration=2.88038369 podStartE2EDuration="2.88038369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:45.878873062 +0000 UTC m=+6101.212086631" watchObservedRunningTime="2026-11-03 17:56:45.88038369 +0000 UTC m=+6101.213597260" Nov 03 17:56:46.838575 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:46.838541 6001 generic.go:356] "Generic (PLEG): container finished" podID="cd241444-9fa1-4750-9cd5-23ea00f38d71" containerID="99da88aedf01df45a1657333f2d0407ed387972b302e706de66e52a87fe671aa" exitCode=0 Nov 03 17:56:46.839013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:46.838611 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" event={"ID":"cd241444-9fa1-4750-9cd5-23ea00f38d71","Type":"ContainerDied","Data":"99da88aedf01df45a1657333f2d0407ed387972b302e706de66e52a87fe671aa"} Nov 03 17:56:47.979616 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:47.979592 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:48.046008 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.045967 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\") pod \"cd241444-9fa1-4750-9cd5-23ea00f38d71\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " Nov 03 17:56:48.046220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.046057 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\") pod \"cd241444-9fa1-4750-9cd5-23ea00f38d71\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " Nov 03 17:56:48.046220 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.046101 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\" (UniqueName: \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\") pod \"cd241444-9fa1-4750-9cd5-23ea00f38d71\" (UID: \"cd241444-9fa1-4750-9cd5-23ea00f38d71\") " Nov 03 17:56:48.046439 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.046412 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca" pod "cd241444-9fa1-4750-9cd5-23ea00f38d71" (UID: "cd241444-9fa1-4750-9cd5-23ea00f38d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:56:48.048291 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.048262 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca" pod "cd241444-9fa1-4750-9cd5-23ea00f38d71" (UID: "cd241444-9fa1-4750-9cd5-23ea00f38d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:56:48.048403 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.048307 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht" pod "cd241444-9fa1-4750-9cd5-23ea00f38d71" (UID: "cd241444-9fa1-4750-9cd5-23ea00f38d71"). InnerVolumeSpecName "kube-api-access-8d8ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:56:48.147038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.146954 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:48.147038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.146984 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd241444-9fa1-4750-9cd5-23ea00f38d71-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:48.147038 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.146993 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d8ht\" (UniqueName: \"kubernetes.io/projected/cd241444-9fa1-4750-9cd5-23ea00f38d71-kube-api-access-8d8ht\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:56:48.850857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.850825 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" Nov 03 17:56:48.850857 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.850848 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79" event={"ID":"cd241444-9fa1-4750-9cd5-23ea00f38d71","Type":"ContainerDied","Data":"c5091f179ab2cd67517f1139b30bf609e3428c790bbffe8a8c3fcd8c69390c40"} Nov 03 17:56:48.851077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:48.850902 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5091f179ab2cd67517f1139b30bf609e3428c790bbffe8a8c3fcd8c69390c40" Nov 03 17:56:52.250463 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:56:52.250413 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:56:59.028613 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:59.028579 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2"] Nov 03 17:56:59.040973 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:56:59.040943 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-pm4n2"] Nov 03 17:57:00.788184 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:00.788143 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff0d2c8-48f7-45fe-ad19-30358fe5f582" path="/var/lib/kubelet/pods/2ff0d2c8-48f7-45fe-ad19-30358fe5f582/volumes" Nov 03 17:57:09.030292 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:09.030255 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2"] Nov 03 17:57:09.042521 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:09.042496 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-wtsf2"] Nov 03 17:57:10.790339 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:10.790301 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85430cc0-adbc-4e32-8182-ee4c60984317" path="/var/lib/kubelet/pods/85430cc0-adbc-4e32-8182-ee4c60984317/volumes" Nov 03 17:57:14.212927 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.212887 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f"] Nov 03 17:57:14.213380 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.213359 6001 cpu_manager.go:414] "containerMap: removing container" podUID="cd241444-9fa1-4750-9cd5-23ea00f38d71" containerName="revoke-keys" Nov 03 17:57:14.213427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.213376 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd241444-9fa1-4750-9cd5-23ea00f38d71" containerName="revoke-keys" Nov 03 17:57:14.213510 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.213492 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd241444-9fa1-4750-9cd5-23ea00f38d71" containerName="revoke-keys" Nov 03 17:57:14.218171 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.218148 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.239540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.239509 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f"] Nov 03 17:57:14.249808 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.249780 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:14.249996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.249862 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:57:14.249996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.249885 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:14.249996 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.249907 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:14.351535 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.351494 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.351774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.351683 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kznwv\" (UniqueName: \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.351774 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.351725 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.452532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.452497 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kznwv\" (UniqueName: \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.452532 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.452534 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.452772 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.452595 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.453382 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.453355 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.455115 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.455094 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.477337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.477305 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kznwv\" (UniqueName: \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\") pod \"maas-api-revoke-sub-e71c6725-8cm8f\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.532196 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.532157 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:14.837331 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:14.837294 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f"] Nov 03 17:57:14.838669 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:57:14.838638 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb327586_8632_4578_8efe_cd725ba5ee93.slice/crio-3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b WatchSource:0}: Error finding container 3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b: Status 404 returned error can't find the container with id 3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b Nov 03 17:57:15.022400 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:15.022367 6001 scope.go:122] "RemoveContainer" containerID="cfa015f5beb8881e6dedcd41afb72cc45fa6f377b1bb3e775a35560dcf37900f" Nov 03 17:57:15.031231 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:15.031212 6001 scope.go:122] "RemoveContainer" containerID="7090bda958a8ff9d81ee5dff96cb9c9f8d2054d73e9d8fe7cfc6e3780f7ed6ae" Nov 03 17:57:15.040289 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:15.040269 6001 scope.go:122] "RemoveContainer" containerID="f37d2c0ae91aabdc6f8adff873d33082fe2bacc68cb640192265c5159bfb60e7" Nov 03 17:57:15.041760 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:15.041733 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" event={"ID":"bb327586-8632-4578-8efe-cd725ba5ee93","Type":"ContainerStarted","Data":"3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b"} Nov 03 17:57:15.049608 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:15.049592 6001 scope.go:122] "RemoveContainer" containerID="5bcb40ec16b1236e67ef44c359e7afae79c6ce2c24645ce8c3da8afcd0a21146" Nov 03 17:57:16.050072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:16.049977 6001 generic.go:356] "Generic (PLEG): container finished" podID="bb327586-8632-4578-8efe-cd725ba5ee93" containerID="7b00d21a9c1c8f4623a03ae9cf0fb3709b6f966bee1d93db1b2c62abc167e492" exitCode=0 Nov 03 17:57:16.050072 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:16.050018 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" event={"ID":"bb327586-8632-4578-8efe-cd725ba5ee93","Type":"ContainerDied","Data":"7b00d21a9c1c8f4623a03ae9cf0fb3709b6f966bee1d93db1b2c62abc167e492"} Nov 03 17:57:17.199287 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.199264 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:17.385652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.385556 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\" (UniqueName: \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\") pod \"bb327586-8632-4578-8efe-cd725ba5ee93\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " Nov 03 17:57:17.385652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.385594 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\") pod \"bb327586-8632-4578-8efe-cd725ba5ee93\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " Nov 03 17:57:17.385917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.385702 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\") pod \"bb327586-8632-4578-8efe-cd725ba5ee93\" (UID: \"bb327586-8632-4578-8efe-cd725ba5ee93\") " Nov 03 17:57:17.386047 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.386023 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca" pod "bb327586-8632-4578-8efe-cd725ba5ee93" (UID: "bb327586-8632-4578-8efe-cd725ba5ee93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:17.387859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.387832 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca" pod "bb327586-8632-4578-8efe-cd725ba5ee93" (UID: "bb327586-8632-4578-8efe-cd725ba5ee93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:17.387859 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.387853 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv" pod "bb327586-8632-4578-8efe-cd725ba5ee93" (UID: "bb327586-8632-4578-8efe-cd725ba5ee93"). InnerVolumeSpecName "kube-api-access-kznwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:17.487334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.487287 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kznwv\" (UniqueName: \"kubernetes.io/projected/bb327586-8632-4578-8efe-cd725ba5ee93-kube-api-access-kznwv\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:17.487334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.487327 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:17.487334 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:17.487343 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb327586-8632-4578-8efe-cd725ba5ee93-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:18.062266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:18.062236 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" Nov 03 17:57:18.062266 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:18.062248 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f" event={"ID":"bb327586-8632-4578-8efe-cd725ba5ee93","Type":"ContainerDied","Data":"3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b"} Nov 03 17:57:18.062514 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:18.062293 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1f9adf21eefb1ef24b41a26b114ce4a036f5fb56fe0ce8ec740c18e320419b" Nov 03 17:57:22.249709 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:57:22.249675 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:57:24.564263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.564228 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76"] Nov 03 17:57:24.564807 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.564788 6001 cpu_manager.go:414] "containerMap: removing container" podUID="bb327586-8632-4578-8efe-cd725ba5ee93" containerName="revoke-keys" Nov 03 17:57:24.564870 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.564808 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb327586-8632-4578-8efe-cd725ba5ee93" containerName="revoke-keys" Nov 03 17:57:24.564918 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.564910 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb327586-8632-4578-8efe-cd725ba5ee93" containerName="revoke-keys" Nov 03 17:57:24.568151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.568129 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.587765 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.587729 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:24.587970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.587840 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:57:24.600686 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.600651 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:24.600854 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.600705 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:24.602977 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.602950 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76"] Nov 03 17:57:24.653385 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.653347 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.653597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.653403 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.653597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.653516 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79vt\" (UniqueName: \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.755151 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.755108 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79vt\" (UniqueName: \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.755393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.755240 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.755393 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.755267 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.755947 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.755921 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.757748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.757721 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.783371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.783336 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79vt\" (UniqueName: \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\") pod \"maas-api-revoke-sub-55d5ae6a-bdn76\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:24.884063 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:24.883969 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:25.185013 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:25.184980 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76"] Nov 03 17:57:25.186046 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:57:25.186014 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93f4803_6fa1_4108_96a4_c2ea1faa7609.slice/crio-4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183 WatchSource:0}: Error finding container 4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183: Status 404 returned error can't find the container with id 4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183 Nov 03 17:57:26.109978 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:26.109939 6001 generic.go:356] "Generic (PLEG): container finished" podID="c93f4803-6fa1-4108-96a4-c2ea1faa7609" containerID="d5352254a88640d02169f4ab1e64ad685c4d83e5e028a4e3dcec74581f916731" exitCode=0 Nov 03 17:57:26.110416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:26.110027 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" event={"ID":"c93f4803-6fa1-4108-96a4-c2ea1faa7609","Type":"ContainerDied","Data":"d5352254a88640d02169f4ab1e64ad685c4d83e5e028a4e3dcec74581f916731"} Nov 03 17:57:26.110416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:26.110083 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" event={"ID":"c93f4803-6fa1-4108-96a4-c2ea1faa7609","Type":"ContainerStarted","Data":"4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183"} Nov 03 17:57:27.253584 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.253558 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:27.280139 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.280106 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\") pod \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " Nov 03 17:57:27.280317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.280206 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\" (UniqueName: \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\") pod \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " Nov 03 17:57:27.280317 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.280266 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\") pod \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\" (UID: \"c93f4803-6fa1-4108-96a4-c2ea1faa7609\") " Nov 03 17:57:27.280627 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.280597 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca" pod "c93f4803-6fa1-4108-96a4-c2ea1faa7609" (UID: "c93f4803-6fa1-4108-96a4-c2ea1faa7609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:27.282635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.282610 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca" pod "c93f4803-6fa1-4108-96a4-c2ea1faa7609" (UID: "c93f4803-6fa1-4108-96a4-c2ea1faa7609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:27.282732 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.282661 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt" pod "c93f4803-6fa1-4108-96a4-c2ea1faa7609" (UID: "c93f4803-6fa1-4108-96a4-c2ea1faa7609"). InnerVolumeSpecName "kube-api-access-z79vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:27.381754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.381658 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:27.381754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.381693 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79vt\" (UniqueName: \"kubernetes.io/projected/c93f4803-6fa1-4108-96a4-c2ea1faa7609-kube-api-access-z79vt\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:27.381754 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:27.381703 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c93f4803-6fa1-4108-96a4-c2ea1faa7609-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:28.121634 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:28.121592 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" event={"ID":"c93f4803-6fa1-4108-96a4-c2ea1faa7609","Type":"ContainerDied","Data":"4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183"} Nov 03 17:57:28.121792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:28.121643 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9be988d85a5fa43ff85b6d33ce6e89c8a55cae9937c5501989839944a05183" Nov 03 17:57:28.121792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:28.121606 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76" Nov 03 17:57:46.999015 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:46.998972 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2"] Nov 03 17:57:46.999560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:46.999418 6001 cpu_manager.go:414] "containerMap: removing container" podUID="c93f4803-6fa1-4108-96a4-c2ea1faa7609" containerName="revoke-keys" Nov 03 17:57:46.999560 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:46.999433 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c93f4803-6fa1-4108-96a4-c2ea1faa7609" containerName="revoke-keys" Nov 03 17:57:47.000332 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.000297 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="c93f4803-6fa1-4108-96a4-c2ea1faa7609" containerName="revoke-keys" Nov 03 17:57:47.006026 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.005988 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.024487 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.024455 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2"] Nov 03 17:57:47.035162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.034718 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:57:47.035162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.034884 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:57:47.035162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.034916 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:57:47.035162 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.034970 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:57:47.115124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.115088 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.115339 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.115156 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkczk\" (UniqueName: \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.115339 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.115257 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.216412 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.216374 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.216635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.216437 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkczk\" (UniqueName: \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.216635 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.216512 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.217078 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.217053 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.219644 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.219611 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.248883 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.248844 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkczk\" (UniqueName: \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\") pod \"maas-api-revoke-sub-01948d0b-464q2\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.323791 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.323686 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:47.629036 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:47.629006 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2"] Nov 03 17:57:47.630278 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:57:47.630245 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6566161b_4221_4cb2_b878_10ba73c722f8.slice/crio-73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83 WatchSource:0}: Error finding container 73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83: Status 404 returned error can't find the container with id 73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83 Nov 03 17:57:48.245430 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:48.245387 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" event={"ID":"6566161b-4221-4cb2-b878-10ba73c722f8","Type":"ContainerStarted","Data":"d3797fa3cd8bd8d2b6e33e3b8c84bebe92f13d192a323f8ff81d899281b1e40c"} Nov 03 17:57:48.245917 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:48.245491 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" event={"ID":"6566161b-4221-4cb2-b878-10ba73c722f8","Type":"ContainerStarted","Data":"73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83"} Nov 03 17:57:48.289040 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:48.288985 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" podStartSLOduration=2.288970217 podStartE2EDuration="2.288970217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:48.287824525 +0000 UTC m=+6163.621038096" watchObservedRunningTime="2026-11-03 17:57:48.288970217 +0000 UTC m=+6163.622183786" Nov 03 17:57:49.253055 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:49.253013 6001 generic.go:356] "Generic (PLEG): container finished" podID="6566161b-4221-4cb2-b878-10ba73c722f8" containerID="d3797fa3cd8bd8d2b6e33e3b8c84bebe92f13d192a323f8ff81d899281b1e40c" exitCode=0 Nov 03 17:57:49.253528 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:49.253078 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" event={"ID":"6566161b-4221-4cb2-b878-10ba73c722f8","Type":"ContainerDied","Data":"d3797fa3cd8bd8d2b6e33e3b8c84bebe92f13d192a323f8ff81d899281b1e40c"} Nov 03 17:57:50.401931 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.401898 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:50.555612 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.555509 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\") pod \"6566161b-4221-4cb2-b878-10ba73c722f8\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " Nov 03 17:57:50.555828 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.555730 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\") pod \"6566161b-4221-4cb2-b878-10ba73c722f8\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " Nov 03 17:57:50.555906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.555830 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\" (UniqueName: \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\") pod \"6566161b-4221-4cb2-b878-10ba73c722f8\" (UID: \"6566161b-4221-4cb2-b878-10ba73c722f8\") " Nov 03 17:57:50.555906 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.555881 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca" pod "6566161b-4221-4cb2-b878-10ba73c722f8" (UID: "6566161b-4221-4cb2-b878-10ba73c722f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:50.556177 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.556159 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:50.557932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.557898 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca" pod "6566161b-4221-4cb2-b878-10ba73c722f8" (UID: "6566161b-4221-4cb2-b878-10ba73c722f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:50.558030 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.558002 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk" pod "6566161b-4221-4cb2-b878-10ba73c722f8" (UID: "6566161b-4221-4cb2-b878-10ba73c722f8"). InnerVolumeSpecName "kube-api-access-mkczk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:50.657416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.657379 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkczk\" (UniqueName: \"kubernetes.io/projected/6566161b-4221-4cb2-b878-10ba73c722f8-kube-api-access-mkczk\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:50.657416 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:50.657412 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6566161b-4221-4cb2-b878-10ba73c722f8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:57:51.265167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:51.265134 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" Nov 03 17:57:51.265357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:51.265236 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2" event={"ID":"6566161b-4221-4cb2-b878-10ba73c722f8","Type":"ContainerDied","Data":"73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83"} Nov 03 17:57:51.265357 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:57:51.265289 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73755c579ce1db7b51e112705bf70fbecd5e24f95760e62e88e5f9f4e23afb83" Nov 03 17:57:52.249866 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:57:52.249830 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:58:15.088354 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:15.088317 6001 scope.go:122] "RemoveContainer" containerID="41fc9a21cab69175b04376dca1e173aec4f9a3abad87fde2b7cff361aad9ad67" Nov 03 17:58:15.099690 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:15.099661 6001 scope.go:122] "RemoveContainer" containerID="5d50d4542ac3ce759741db3673aa856b0fd25a22aa4fe24500dc1340a8b0478e" Nov 03 17:58:15.108642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:15.108616 6001 scope.go:122] "RemoveContainer" containerID="42990cf3cf8ab74e0cfa9344dd16247eba211fb80204fc4d5ed700ca861090fd" Nov 03 17:58:19.865597 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.864461 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf"] Nov 03 17:58:19.866281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.865569 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6566161b-4221-4cb2-b878-10ba73c722f8" containerName="revoke-keys" Nov 03 17:58:19.866281 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.866285 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6566161b-4221-4cb2-b878-10ba73c722f8" containerName="revoke-keys" Nov 03 17:58:19.866652 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.866634 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6566161b-4221-4cb2-b878-10ba73c722f8" containerName="revoke-keys" Nov 03 17:58:19.875589 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.875556 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:19.888981 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.888933 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf"] Nov 03 17:58:19.894333 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.894310 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:19.899571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.899543 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:19.899571 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.899565 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:19.899802 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.899667 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:58:19.942108 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.942071 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:19.942294 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.942126 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwtq4\" (UniqueName: \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:19.942389 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:19.942286 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.043520 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.043475 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.043755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.043545 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwtq4\" (UniqueName: \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.043755 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.043642 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.044282 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.044254 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.046186 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.046154 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.068813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.068780 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwtq4\" (UniqueName: \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\") pod \"maas-api-revoke-sub-990be415-6xhvf\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.202546 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.202410 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:20.511792 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:20.511758 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf"] Nov 03 17:58:20.512288 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:58:20.512255 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3349b5e_c4a1_4e9a_84a8_7a4b0f8ca678.slice/crio-9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a WatchSource:0}: Error finding container 9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a: Status 404 returned error can't find the container with id 9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a Nov 03 17:58:21.457258 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:21.457229 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" event={"ID":"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678","Type":"ContainerStarted","Data":"7d635da4d7022b1438930da1db4b3522c1eed1c1209df18fb0bf0e0abe230bd8"} Nov 03 17:58:21.457681 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:21.457280 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" event={"ID":"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678","Type":"ContainerStarted","Data":"9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a"} Nov 03 17:58:22.251138 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:58:22.251103 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:58:22.464941 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.464902 6001 generic.go:356] "Generic (PLEG): container finished" podID="e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" containerID="7d635da4d7022b1438930da1db4b3522c1eed1c1209df18fb0bf0e0abe230bd8" exitCode=0 Nov 03 17:58:22.465421 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.464994 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" event={"ID":"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678","Type":"ContainerDied","Data":"7d635da4d7022b1438930da1db4b3522c1eed1c1209df18fb0bf0e0abe230bd8"} Nov 03 17:58:22.606401 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.606369 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:22.669888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.669855 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\") pod \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " Nov 03 17:58:22.670113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.669927 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\" (UniqueName: \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\") pod \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " Nov 03 17:58:22.670113 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.670085 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\") pod \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\" (UID: \"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678\") " Nov 03 17:58:22.670330 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.670305 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca" pod "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" (UID: "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:22.670569 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.670549 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:22.672222 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.672194 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4" pod "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" (UID: "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678"). InnerVolumeSpecName "kube-api-access-lwtq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:22.672378 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.672212 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca" pod "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" (UID: "e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:58:22.771368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.771258 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwtq4\" (UniqueName: \"kubernetes.io/projected/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-kube-api-access-lwtq4\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:22.771368 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:22.771307 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:23.471694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:23.471660 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" Nov 03 17:58:23.471694 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:23.471677 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-990be415-6xhvf" event={"ID":"e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678","Type":"ContainerDied","Data":"9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a"} Nov 03 17:58:23.472211 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:23.471739 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9839fa3d723f4e7184d32672a3e17d8547a27f9f064756018e54fd0ec4fe259a" Nov 03 17:58:30.213711 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.213672 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv"] Nov 03 17:58:30.214137 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.214123 6001 cpu_manager.go:414] "containerMap: removing container" podUID="e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" containerName="revoke-keys" Nov 03 17:58:30.214190 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.214138 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" containerName="revoke-keys" Nov 03 17:58:30.214257 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.214246 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678" containerName="revoke-keys" Nov 03 17:58:30.217893 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.217864 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.237770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.237740 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:30.250312 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.250279 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:58:30.252526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.252473 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:30.252526 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.252513 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:30.252759 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.252663 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv"] Nov 03 17:58:30.347869 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.347822 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghdb\" (UniqueName: \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.348077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.347876 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.348132 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.348069 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.448756 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.448715 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.448959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.448795 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghdb\" (UniqueName: \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.448959 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.448820 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.449536 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.449510 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.451263 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.451242 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.472937 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.472907 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghdb\" (UniqueName: \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\") pod \"maas-api-revoke-sub-71a86c5d-7tghv\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.533102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.533065 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:30.839155 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:30.839114 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv"] Nov 03 17:58:30.840327 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:58:30.840300 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5d4562_808a_4c15_875d_683e2b5cd59c.slice/crio-40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590 WatchSource:0}: Error finding container 40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590: Status 404 returned error can't find the container with id 40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590 Nov 03 17:58:31.515888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:31.515846 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" event={"ID":"1e5d4562-808a-4c15-875d-683e2b5cd59c","Type":"ContainerStarted","Data":"e805e7e5af0fc7e011d002eadbf8ec2c365104c6f3dee4ddaf29a7c8b4b886d0"} Nov 03 17:58:31.516429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:31.515916 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" event={"ID":"1e5d4562-808a-4c15-875d-683e2b5cd59c","Type":"ContainerStarted","Data":"40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590"} Nov 03 17:58:31.558956 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:31.558899 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" podStartSLOduration=1.55888545 podStartE2EDuration="1.55888545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:31.557878212 +0000 UTC m=+6206.891091782" watchObservedRunningTime="2026-11-03 17:58:31.55888545 +0000 UTC m=+6206.892099018" Nov 03 17:58:32.522899 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:32.522863 6001 generic.go:356] "Generic (PLEG): container finished" podID="1e5d4562-808a-4c15-875d-683e2b5cd59c" containerID="e805e7e5af0fc7e011d002eadbf8ec2c365104c6f3dee4ddaf29a7c8b4b886d0" exitCode=0 Nov 03 17:58:32.523337 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:32.522949 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" event={"ID":"1e5d4562-808a-4c15-875d-683e2b5cd59c","Type":"ContainerDied","Data":"e805e7e5af0fc7e011d002eadbf8ec2c365104c6f3dee4ddaf29a7c8b4b886d0"} Nov 03 17:58:33.667534 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.667506 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:33.780142 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.780041 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\" (UniqueName: \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\") pod \"1e5d4562-808a-4c15-875d-683e2b5cd59c\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " Nov 03 17:58:33.780336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.780173 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\") pod \"1e5d4562-808a-4c15-875d-683e2b5cd59c\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " Nov 03 17:58:33.780336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.780207 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\") pod \"1e5d4562-808a-4c15-875d-683e2b5cd59c\" (UID: \"1e5d4562-808a-4c15-875d-683e2b5cd59c\") " Nov 03 17:58:33.780631 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.780610 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca" pod "1e5d4562-808a-4c15-875d-683e2b5cd59c" (UID: "1e5d4562-808a-4c15-875d-683e2b5cd59c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:58:33.782371 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.782338 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb" pod "1e5d4562-808a-4c15-875d-683e2b5cd59c" (UID: "1e5d4562-808a-4c15-875d-683e2b5cd59c"). InnerVolumeSpecName "kube-api-access-wghdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:58:33.782543 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.782454 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca" pod "1e5d4562-808a-4c15-875d-683e2b5cd59c" (UID: "1e5d4562-808a-4c15-875d-683e2b5cd59c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:58:33.881757 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.881715 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:33.881940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.881808 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5d4562-808a-4c15-875d-683e2b5cd59c-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:33.881940 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:33.881843 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghdb\" (UniqueName: \"kubernetes.io/projected/1e5d4562-808a-4c15-875d-683e2b5cd59c-kube-api-access-wghdb\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:58:34.033578 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.033481 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw"] Nov 03 17:58:34.035789 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.035763 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1002a66-9cvvw"] Nov 03 17:58:34.535541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.535505 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" Nov 03 17:58:34.535747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.535511 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71a86c5d-7tghv" event={"ID":"1e5d4562-808a-4c15-875d-683e2b5cd59c","Type":"ContainerDied","Data":"40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590"} Nov 03 17:58:34.535747 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.535643 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ef9afd2af08db28413f875d6f0fcb422c8f2f2b132c2c209e00ec405e18590" Nov 03 17:58:34.787552 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:34.787467 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8645435-031d-41cf-a879-f25ff0d2ace7" path="/var/lib/kubelet/pods/f8645435-031d-41cf-a879-f25ff0d2ace7/volumes" Nov 03 17:58:52.249396 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:58:52.249318 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:58:57.993024 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:57.992990 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm"] Nov 03 17:58:57.993423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:57.993411 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1e5d4562-808a-4c15-875d-683e2b5cd59c" containerName="revoke-keys" Nov 03 17:58:57.993423 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:57.993422 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5d4562-808a-4c15-875d-683e2b5cd59c" containerName="revoke-keys" Nov 03 17:58:57.993544 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:57.993531 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5d4562-808a-4c15-875d-683e2b5cd59c" containerName="revoke-keys" Nov 03 17:58:57.996540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:57.996523 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.011949 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.011929 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:58:58.012148 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.012132 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:58:58.024621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.024597 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:58:58.024621 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.024600 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:58:58.026588 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.026567 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm"] Nov 03 17:58:58.140954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.140898 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.140954 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.140966 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrzrr\" (UniqueName: \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.141241 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.141069 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.241923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.241884 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrzrr\" (UniqueName: \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.241923 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.241925 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.242159 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.242038 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.242821 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.242799 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.244481 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.244415 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.272127 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.272085 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrzrr\" (UniqueName: \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\") pod \"maas-api-revoke-sub-6c9144af-79pjm\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.309572 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.309532 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:58:58.620540 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.620509 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm"] Nov 03 17:58:58.620995 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:58:58.620970 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05ee6e6_7abb_406a_bee8_713c61afe7b1.slice/crio-830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd WatchSource:0}: Error finding container 830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd: Status 404 returned error can't find the container with id 830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd Nov 03 17:58:58.763398 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:58.763362 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" event={"ID":"d05ee6e6-7abb-406a-bee8-713c61afe7b1","Type":"ContainerStarted","Data":"830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd"} Nov 03 17:58:59.770140 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:59.770102 6001 generic.go:356] "Generic (PLEG): container finished" podID="d05ee6e6-7abb-406a-bee8-713c61afe7b1" containerID="c6ccb8a98ab179d5d4cb7bc694ee712bdabbf49b6d471a889e7891d693a50e8e" exitCode=0 Nov 03 17:58:59.770581 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:58:59.770189 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" event={"ID":"d05ee6e6-7abb-406a-bee8-713c61afe7b1","Type":"ContainerDied","Data":"c6ccb8a98ab179d5d4cb7bc694ee712bdabbf49b6d471a889e7891d693a50e8e"} Nov 03 17:59:00.931077 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:00.931044 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:59:01.072384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.072297 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\" (UniqueName: \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\") pod \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " Nov 03 17:59:01.072384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.072343 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\") pod \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " Nov 03 17:59:01.072384 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.072364 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\") pod \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\" (UID: \"d05ee6e6-7abb-406a-bee8-713c61afe7b1\") " Nov 03 17:59:01.072770 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.072746 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca" pod "d05ee6e6-7abb-406a-bee8-713c61afe7b1" (UID: "d05ee6e6-7abb-406a-bee8-713c61afe7b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:01.072955 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.072930 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:01.074626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.074602 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr" pod "d05ee6e6-7abb-406a-bee8-713c61afe7b1" (UID: "d05ee6e6-7abb-406a-bee8-713c61afe7b1"). InnerVolumeSpecName "kube-api-access-jrzrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:01.074626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.074613 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca" pod "d05ee6e6-7abb-406a-bee8-713c61afe7b1" (UID: "d05ee6e6-7abb-406a-bee8-713c61afe7b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:59:01.173873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.173832 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrzrr\" (UniqueName: \"kubernetes.io/projected/d05ee6e6-7abb-406a-bee8-713c61afe7b1-kube-api-access-jrzrr\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:01.173873 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.173866 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d05ee6e6-7abb-406a-bee8-713c61afe7b1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:01.782054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.782024 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" Nov 03 17:59:01.782054 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.782042 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9144af-79pjm" event={"ID":"d05ee6e6-7abb-406a-bee8-713c61afe7b1","Type":"ContainerDied","Data":"830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd"} Nov 03 17:59:01.782304 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:01.782083 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="830ca1b4ec77e7bcda1f458fc09db09b4b75af05941b0308646e354a35171dfd" Nov 03 17:59:08.332661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.332622 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n"] Nov 03 17:59:08.333269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.333086 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d05ee6e6-7abb-406a-bee8-713c61afe7b1" containerName="revoke-keys" Nov 03 17:59:08.333269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.333100 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05ee6e6-7abb-406a-bee8-713c61afe7b1" containerName="revoke-keys" Nov 03 17:59:08.333269 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.333219 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05ee6e6-7abb-406a-bee8-713c61afe7b1" containerName="revoke-keys" Nov 03 17:59:08.336568 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.336549 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.342737 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.342635 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.342888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.342809 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.342888 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.342847 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62nzl\" (UniqueName: \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.352340 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.352310 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:59:08.364961 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.364925 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:59:08.368256 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.368232 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n"] Nov 03 17:59:08.390154 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.390121 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:59:08.390344 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.390168 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:59:08.443958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.443921 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.443958 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.443958 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62nzl\" (UniqueName: \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.444194 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.444030 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.444731 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.444699 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.446321 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.446295 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.468243 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.468211 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62nzl\" (UniqueName: \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\") pod \"maas-api-revoke-sub-69c09e8b-wqv8n\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.650314 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.650214 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:08.955370 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:59:08.955341 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8f4aef_6fb9_4cd3_b7d8_ce0c2e63d3df.slice/crio-be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4 WatchSource:0}: Error finding container be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4: Status 404 returned error can't find the container with id be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4 Nov 03 17:59:08.955692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:08.955666 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n"] Nov 03 17:59:09.829087 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:09.829042 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" event={"ID":"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df","Type":"ContainerStarted","Data":"e310adc09f7c23e787a4ae77013376f88b968ad045b314af32a1948e9a75f9c2"} Nov 03 17:59:09.829545 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:09.829111 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" event={"ID":"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df","Type":"ContainerStarted","Data":"be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4"} Nov 03 17:59:09.873164 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:09.873105 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" podStartSLOduration=1.873089737 podStartE2EDuration="1.873089737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:09.872302568 +0000 UTC m=+6245.205516149" watchObservedRunningTime="2026-11-03 17:59:09.873089737 +0000 UTC m=+6245.206303363" Nov 03 17:59:10.837212 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:10.837165 6001 generic.go:356] "Generic (PLEG): container finished" podID="4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" containerID="e310adc09f7c23e787a4ae77013376f88b968ad045b314af32a1948e9a75f9c2" exitCode=0 Nov 03 17:59:10.837664 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:10.837308 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" event={"ID":"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df","Type":"ContainerDied","Data":"e310adc09f7c23e787a4ae77013376f88b968ad045b314af32a1948e9a75f9c2"} Nov 03 17:59:12.234982 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.234951 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:12.341114 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.341079 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\") pod \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " Nov 03 17:59:12.341336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.341174 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\" (UniqueName: \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\") pod \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " Nov 03 17:59:12.341336 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.341209 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\") pod \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\" (UID: \"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df\") " Nov 03 17:59:12.342032 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.341999 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca" pod "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" (UID: "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:12.364469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.354582 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca" pod "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" (UID: "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:59:12.364469 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.355477 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl" pod "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" (UID: "4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df"). InnerVolumeSpecName "kube-api-access-62nzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:12.442736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.442634 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62nzl\" (UniqueName: \"kubernetes.io/projected/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-kube-api-access-62nzl\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:12.442736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.442675 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:12.442736 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.442687 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:12.849559 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.849528 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" Nov 03 17:59:12.849748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.849529 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69c09e8b-wqv8n" event={"ID":"4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df","Type":"ContainerDied","Data":"be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4"} Nov 03 17:59:12.849748 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:12.849660 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be31aa8f5ee3825c5a8b26c1076bd69ab20f49a9fbce76e253583d7d372141f4" Nov 03 17:59:15.137553 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:15.137522 6001 scope.go:122] "RemoveContainer" containerID="94b922408a21b254762fc28a15070ce64fa8e9c333888269289ce87d54835913" Nov 03 17:59:22.251669 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:59:22.251629 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:59:32.997573 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:32.997533 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:59:32.998320 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:32.998285 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.820692 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.820656 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644"] Nov 03 17:59:40.821124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.821086 6001 cpu_manager.go:414] "containerMap: removing container" podUID="4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" containerName="revoke-keys" Nov 03 17:59:40.821124 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.821097 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" containerName="revoke-keys" Nov 03 17:59:40.821213 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.821180 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df" containerName="revoke-keys" Nov 03 17:59:40.824605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.824582 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.853275 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.853243 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:59:40.853522 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.853396 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:59:40.853653 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.853590 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:59:40.853777 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.853679 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:59:40.855908 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.855887 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644"] Nov 03 17:59:40.856375 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.856349 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.856429 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.856408 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll7hf\" (UniqueName: \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.856607 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.856585 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.957626 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.957585 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.957813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.957635 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll7hf\" (UniqueName: \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.957813 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.957666 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.958271 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.958239 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.960003 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.959980 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:40.982033 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:40.981994 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll7hf\" (UniqueName: \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\") pod \"maas-api-revoke-sub-84d39b4f-9x644\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:41.139202 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:41.139092 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:41.450456 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:41.450411 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644"] Nov 03 17:59:41.450677 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:59:41.450650 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6cae8c_3e17_441a_80fc_07c04dee42e8.slice/crio-0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418 WatchSource:0}: Error finding container 0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418: Status 404 returned error can't find the container with id 0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418 Nov 03 17:59:42.057512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:42.057404 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" event={"ID":"fe6cae8c-3e17-441a-80fc-07c04dee42e8","Type":"ContainerStarted","Data":"8f427df4bca520aff978c0e173c7d341956cf25b59085db1ae0b4c045462f01c"} Nov 03 17:59:42.057512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:42.057476 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" event={"ID":"fe6cae8c-3e17-441a-80fc-07c04dee42e8","Type":"ContainerStarted","Data":"0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418"} Nov 03 17:59:42.103661 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:42.103610 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" podStartSLOduration=2.103596771 podStartE2EDuration="2.103596771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:42.101700013 +0000 UTC m=+6277.434913582" watchObservedRunningTime="2026-11-03 17:59:42.103596771 +0000 UTC m=+6277.436810339" Nov 03 17:59:43.064583 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:43.064552 6001 generic.go:356] "Generic (PLEG): container finished" podID="fe6cae8c-3e17-441a-80fc-07c04dee42e8" containerID="8f427df4bca520aff978c0e173c7d341956cf25b59085db1ae0b4c045462f01c" exitCode=0 Nov 03 17:59:43.065007 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:43.064632 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" event={"ID":"fe6cae8c-3e17-441a-80fc-07c04dee42e8","Type":"ContainerDied","Data":"8f427df4bca520aff978c0e173c7d341956cf25b59085db1ae0b4c045462f01c"} Nov 03 17:59:44.220463 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.220415 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:44.295250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.295211 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\" (UniqueName: \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\") pod \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " Nov 03 17:59:44.295250 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.295257 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\") pod \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " Nov 03 17:59:44.295512 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.295276 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\") pod \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\" (UID: \"fe6cae8c-3e17-441a-80fc-07c04dee42e8\") " Nov 03 17:59:44.295750 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.295716 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca" pod "fe6cae8c-3e17-441a-80fc-07c04dee42e8" (UID: "fe6cae8c-3e17-441a-80fc-07c04dee42e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:44.297503 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.297480 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf" pod "fe6cae8c-3e17-441a-80fc-07c04dee42e8" (UID: "fe6cae8c-3e17-441a-80fc-07c04dee42e8"). InnerVolumeSpecName "kube-api-access-ll7hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:44.297646 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.297618 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca" pod "fe6cae8c-3e17-441a-80fc-07c04dee42e8" (UID: "fe6cae8c-3e17-441a-80fc-07c04dee42e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:59:44.396427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.396323 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll7hf\" (UniqueName: \"kubernetes.io/projected/fe6cae8c-3e17-441a-80fc-07c04dee42e8-kube-api-access-ll7hf\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:44.396427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.396360 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:44.396427 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:44.396376 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe6cae8c-3e17-441a-80fc-07c04dee42e8-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:45.077092 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:45.077012 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" Nov 03 17:59:45.077265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:45.077019 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84d39b4f-9x644" event={"ID":"fe6cae8c-3e17-441a-80fc-07c04dee42e8","Type":"ContainerDied","Data":"0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418"} Nov 03 17:59:45.077265 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:45.077127 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0803480baf7a4fbf05293769eda34b43f06ea0f0d16a340c4c39bdd52ba32418" Nov 03 17:59:49.030128 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:49.030093 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4"] Nov 03 17:59:49.042656 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:49.042612 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-27pw4"] Nov 03 17:59:50.788541 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:50.788497 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0eb223b-120a-420e-898a-b3dbaedd2cc0" path="/var/lib/kubelet/pods/e0eb223b-120a-420e-898a-b3dbaedd2cc0/volumes" Nov 03 17:59:51.145733 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.145652 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt"] Nov 03 17:59:51.146062 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.146050 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fe6cae8c-3e17-441a-80fc-07c04dee42e8" containerName="revoke-keys" Nov 03 17:59:51.146118 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.146063 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe6cae8c-3e17-441a-80fc-07c04dee42e8" containerName="revoke-keys" Nov 03 17:59:51.146167 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.146158 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe6cae8c-3e17-441a-80fc-07c04dee42e8" containerName="revoke-keys" Nov 03 17:59:51.150027 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.150007 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.169693 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.169666 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt"] Nov 03 17:59:51.179932 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.179901 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 17:59:51.180102 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.179996 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 17:59:51.199605 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.199578 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:59:51.199766 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.199620 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:59:51.267525 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.267494 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.267712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.267632 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfwg\" (UniqueName: \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.267712 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.267689 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.368827 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.368787 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.369028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.368878 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.369028 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.368983 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfwg\" (UniqueName: \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.369611 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.369588 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.371230 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.371210 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.393475 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.393433 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfwg\" (UniqueName: \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\") pod \"maas-api-revoke-sub-cb4bbb40-vshjt\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.463219 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.463135 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:51.765288 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:51.765260 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt"] Nov 03 17:59:51.765435 ip-10-0-129-130 kubenswrapper[6001]: W1103 17:59:51.765402 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137500b0_b3aa_43a8_9cef_88ca35eb15b1.slice/crio-e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b WatchSource:0}: Error finding container e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b: Status 404 returned error can't find the container with id e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b Nov 03 17:59:52.118877 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:52.118845 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" event={"ID":"137500b0-b3aa-43a8-9cef-88ca35eb15b1","Type":"ContainerStarted","Data":"be0369b92dc9827eb0106a422cc5d61cad4eb23b986ba8e737082619ed683ffc"} Nov 03 17:59:52.119262 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:52.118896 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" event={"ID":"137500b0-b3aa-43a8-9cef-88ca35eb15b1","Type":"ContainerStarted","Data":"e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b"} Nov 03 17:59:52.166183 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:52.166134 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" podStartSLOduration=1.166120672 podStartE2EDuration="1.166120672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:52.165393569 +0000 UTC m=+6287.498607138" watchObservedRunningTime="2026-11-03 17:59:52.166120672 +0000 UTC m=+6287.499334263" Nov 03 17:59:52.250824 ip-10-0-129-130 kubenswrapper[6001]: E1103 17:59:52.250787 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 17:59:53.125602 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:53.125564 6001 generic.go:356] "Generic (PLEG): container finished" podID="137500b0-b3aa-43a8-9cef-88ca35eb15b1" containerID="be0369b92dc9827eb0106a422cc5d61cad4eb23b986ba8e737082619ed683ffc" exitCode=0 Nov 03 17:59:53.125990 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:53.125650 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" event={"ID":"137500b0-b3aa-43a8-9cef-88ca35eb15b1","Type":"ContainerDied","Data":"be0369b92dc9827eb0106a422cc5d61cad4eb23b986ba8e737082619ed683ffc"} Nov 03 17:59:54.266803 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.266779 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:54.400600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.400510 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\" (UniqueName: \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\") pod \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " Nov 03 17:59:54.400600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.400545 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\") pod \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " Nov 03 17:59:54.400600 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.400571 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\") pod \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\" (UID: \"137500b0-b3aa-43a8-9cef-88ca35eb15b1\") " Nov 03 17:59:54.400916 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.400891 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca" pod "137500b0-b3aa-43a8-9cef-88ca35eb15b1" (UID: "137500b0-b3aa-43a8-9cef-88ca35eb15b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:59:54.402970 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.402945 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg" pod "137500b0-b3aa-43a8-9cef-88ca35eb15b1" (UID: "137500b0-b3aa-43a8-9cef-88ca35eb15b1"). InnerVolumeSpecName "kube-api-access-zrfwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:59:54.403089 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.403015 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca" pod "137500b0-b3aa-43a8-9cef-88ca35eb15b1" (UID: "137500b0-b3aa-43a8-9cef-88ca35eb15b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:59:54.501651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.501607 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:54.501651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.501644 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/137500b0-b3aa-43a8-9cef-88ca35eb15b1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:54.501651 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:54.501653 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfwg\" (UniqueName: \"kubernetes.io/projected/137500b0-b3aa-43a8-9cef-88ca35eb15b1-kube-api-access-zrfwg\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 17:59:55.139642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:55.139544 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" event={"ID":"137500b0-b3aa-43a8-9cef-88ca35eb15b1","Type":"ContainerDied","Data":"e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b"} Nov 03 17:59:55.139642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:55.139585 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb4bbb40-vshjt" Nov 03 17:59:55.139642 ip-10-0-129-130 kubenswrapper[6001]: I1103 17:59:55.139615 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59aa9fae5667c57625c2f09905e9b1239185ee4b5e814e1a85de1099188828b" Nov 03 18:00:00.149753 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.149711 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:00.150349 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.150329 6001 cpu_manager.go:414] "containerMap: removing container" podUID="137500b0-b3aa-43a8-9cef-88ca35eb15b1" containerName="revoke-keys" Nov 03 18:00:00.150422 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.150351 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="137500b0-b3aa-43a8-9cef-88ca35eb15b1" containerName="revoke-keys" Nov 03 18:00:00.150508 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.150431 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="137500b0-b3aa-43a8-9cef-88ca35eb15b1" containerName="revoke-keys" Nov 03 18:00:00.153801 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.153782 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.175282 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.175248 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:00.184556 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.184524 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:00.184731 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.184578 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:00:00.184731 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.184623 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:00.262626 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.262586 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bxj\" (UniqueName: \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.262822 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.262800 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.364192 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.364151 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bxj\" (UniqueName: \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.364388 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.364275 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.364919 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.364899 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.389092 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.389061 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bxj\" (UniqueName: \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\") pod \"maas-api-key-cleanup-29895480-wb572\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.467951 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.467837 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:00.772274 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:00.772232 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:00.772643 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:00.772618 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d7b9af2_4bb9_4c92_915b_b18287e97c6d.slice/crio-208cf7911e7ba4c4c1cf334d7de16b1cd3f4af5129269f8f466686105ee51603 WatchSource:0}: Error finding container 208cf7911e7ba4c4c1cf334d7de16b1cd3f4af5129269f8f466686105ee51603: Status 404 returned error can't find the container with id 208cf7911e7ba4c4c1cf334d7de16b1cd3f4af5129269f8f466686105ee51603 Nov 03 18:00:01.192219 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.192177 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerStarted","Data":"b71121966fd86c3943653fb6536deddfff2a0181f5f555ac820a08c3485dda46"} Nov 03 18:00:01.192724 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.192236 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerStarted","Data":"208cf7911e7ba4c4c1cf334d7de16b1cd3f4af5129269f8f466686105ee51603"} Nov 03 18:00:01.235732 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.235671 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" podStartSLOduration=1.235655467 podStartE2EDuration="1.235655467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:01.23381341 +0000 UTC m=+6296.567026981" watchObservedRunningTime="2026-11-03 18:00:01.235655467 +0000 UTC m=+6296.568869078" Nov 03 18:00:01.462513 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.462401 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq"] Nov 03 18:00:01.466227 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.466209 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.485020 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.484965 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:01.488149 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.488118 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq"] Nov 03 18:00:01.577146 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.577106 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.577334 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.577166 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87zw\" (UniqueName: \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.577334 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.577262 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.678681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.678635 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.678910 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.678697 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87zw\" (UniqueName: \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.678910 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.678730 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.679370 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.679340 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.681240 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.681220 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.703027 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.703001 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87zw\" (UniqueName: \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\") pod \"maas-api-revoke-sub-f6d6c53d-hpzgq\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:01.779994 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:01.779954 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:02.084558 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:02.084519 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq"] Nov 03 18:00:02.084926 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:02.084903 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404cb8bf_0288_4077_b53a_5c72533b41d1.slice/crio-e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e WatchSource:0}: Error finding container e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e: Status 404 returned error can't find the container with id e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e Nov 03 18:00:02.199025 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:02.198991 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" event={"ID":"404cb8bf-0288-4077-b53a-5c72533b41d1","Type":"ContainerStarted","Data":"e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e"} Nov 03 18:00:02.200681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:02.200652 6001 generic.go:356] "Generic (PLEG): container finished" podID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerID="b71121966fd86c3943653fb6536deddfff2a0181f5f555ac820a08c3485dda46" exitCode=60 Nov 03 18:00:02.200836 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:02.200718 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerDied","Data":"b71121966fd86c3943653fb6536deddfff2a0181f5f555ac820a08c3485dda46"} Nov 03 18:00:02.201021 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:02.201007 6001 scope.go:122] "RemoveContainer" containerID="b71121966fd86c3943653fb6536deddfff2a0181f5f555ac820a08c3485dda46" Nov 03 18:00:03.211036 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.210939 6001 generic.go:356] "Generic (PLEG): container finished" podID="404cb8bf-0288-4077-b53a-5c72533b41d1" containerID="dbb3d578ddb94798e37e80161086d73ac240ddb5829f83241503973d50583bb7" exitCode=0 Nov 03 18:00:03.211560 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.211033 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" event={"ID":"404cb8bf-0288-4077-b53a-5c72533b41d1","Type":"ContainerDied","Data":"dbb3d578ddb94798e37e80161086d73ac240ddb5829f83241503973d50583bb7"} Nov 03 18:00:03.213123 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.213096 6001 generic.go:356] "Generic (PLEG): container finished" podID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerID="8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35" exitCode=60 Nov 03 18:00:03.213289 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.213178 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerDied","Data":"8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35"} Nov 03 18:00:03.213289 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.213228 6001 scope.go:122] "RemoveContainer" containerID="b71121966fd86c3943653fb6536deddfff2a0181f5f555ac820a08c3485dda46" Nov 03 18:00:03.213585 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:03.213566 6001 scope.go:122] "RemoveContainer" containerID="8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35" Nov 03 18:00:03.213913 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:00:03.213891 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-wb572_odh-ai-gateway-infra(7d7b9af2-4bb9-4c92-915b-b18287e97c6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" Nov 03 18:00:04.220885 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.220854 6001 scope.go:122] "RemoveContainer" containerID="8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35" Nov 03 18:00:04.221427 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:00:04.221124 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-wb572_odh-ai-gateway-infra(7d7b9af2-4bb9-4c92-915b-b18287e97c6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" Nov 03 18:00:04.370003 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.369974 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:04.510457 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.510395 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\") pod \"404cb8bf-0288-4077-b53a-5c72533b41d1\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " Nov 03 18:00:04.510688 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.510559 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\" (UniqueName: \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\") pod \"404cb8bf-0288-4077-b53a-5c72533b41d1\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " Nov 03 18:00:04.510688 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.510633 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\") pod \"404cb8bf-0288-4077-b53a-5c72533b41d1\" (UID: \"404cb8bf-0288-4077-b53a-5c72533b41d1\") " Nov 03 18:00:04.510832 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.510782 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca" pod "404cb8bf-0288-4077-b53a-5c72533b41d1" (UID: "404cb8bf-0288-4077-b53a-5c72533b41d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:04.511000 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.510980 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:04.512862 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.512837 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca" pod "404cb8bf-0288-4077-b53a-5c72533b41d1" (UID: "404cb8bf-0288-4077-b53a-5c72533b41d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:04.512971 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.512865 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw" pod "404cb8bf-0288-4077-b53a-5c72533b41d1" (UID: "404cb8bf-0288-4077-b53a-5c72533b41d1"). InnerVolumeSpecName "kube-api-access-w87zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:04.611637 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.611588 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87zw\" (UniqueName: \"kubernetes.io/projected/404cb8bf-0288-4077-b53a-5c72533b41d1-kube-api-access-w87zw\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:04.611637 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:04.611629 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/404cb8bf-0288-4077-b53a-5c72533b41d1-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:05.227706 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:05.227675 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" Nov 03 18:00:05.228100 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:05.227698 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6d6c53d-hpzgq" event={"ID":"404cb8bf-0288-4077-b53a-5c72533b41d1","Type":"ContainerDied","Data":"e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e"} Nov 03 18:00:05.228100 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:05.227753 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1390cdaa9e8206189a6485790165dec0105a8acaa6858ed9a1365886eb5e57e" Nov 03 18:00:06.014542 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:06.014513 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 18:00:06.132646 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:06.132615 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c84vd_d16e7255-2933-413f-9114-4b1e2f93d744/ovn-acl-logging/0.log" Nov 03 18:00:11.816829 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.816791 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9"] Nov 03 18:00:11.817289 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.817273 6001 cpu_manager.go:414] "containerMap: removing container" podUID="404cb8bf-0288-4077-b53a-5c72533b41d1" containerName="revoke-keys" Nov 03 18:00:11.817342 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.817289 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404cb8bf-0288-4077-b53a-5c72533b41d1" containerName="revoke-keys" Nov 03 18:00:11.817381 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.817364 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="404cb8bf-0288-4077-b53a-5c72533b41d1" containerName="revoke-keys" Nov 03 18:00:11.820759 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.820737 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:11.839900 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.839865 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:11.854702 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.854670 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9"] Nov 03 18:00:11.984456 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.984405 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:11.984651 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.984555 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7t8j\" (UniqueName: \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:11.984651 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:11.984576 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.085374 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.085263 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.085374 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.085385 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7t8j\" (UniqueName: \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.085618 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.085405 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.086009 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.085972 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.087826 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.087804 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.109824 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.109794 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7t8j\" (UniqueName: \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\") pod \"maas-api-revoke-sub-631ebca0-cv8z9\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.134139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.134100 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:12.436564 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:12.436536 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9"] Nov 03 18:00:12.436943 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:12.436909 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc8ff40_f1a3_42fb_b661_fc216e073c85.slice/crio-e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c WatchSource:0}: Error finding container e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c: Status 404 returned error can't find the container with id e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c Nov 03 18:00:13.275614 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:13.275565 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" event={"ID":"6dc8ff40-f1a3-42fb-b661-fc216e073c85","Type":"ContainerStarted","Data":"d7e83ec799797ca0b80a5d494876b2ad6bea4ca403f262c61dd9cb6b1135ce0d"} Nov 03 18:00:13.276050 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:13.275645 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" event={"ID":"6dc8ff40-f1a3-42fb-b661-fc216e073c85","Type":"ContainerStarted","Data":"e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c"} Nov 03 18:00:13.319799 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:13.319742 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" podStartSLOduration=2.319728264 podStartE2EDuration="2.319728264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:13.318133437 +0000 UTC m=+6308.651347007" watchObservedRunningTime="2026-11-03 18:00:13.319728264 +0000 UTC m=+6308.652941833" Nov 03 18:00:14.313479 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:14.313101 6001 generic.go:356] "Generic (PLEG): container finished" podID="6dc8ff40-f1a3-42fb-b661-fc216e073c85" containerID="d7e83ec799797ca0b80a5d494876b2ad6bea4ca403f262c61dd9cb6b1135ce0d" exitCode=0 Nov 03 18:00:14.313479 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:14.313181 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" event={"ID":"6dc8ff40-f1a3-42fb-b661-fc216e073c85","Type":"ContainerDied","Data":"d7e83ec799797ca0b80a5d494876b2ad6bea4ca403f262c61dd9cb6b1135ce0d"} Nov 03 18:00:15.171352 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.171307 6001 scope.go:122] "RemoveContainer" containerID="f78a130d3ad54405ede2daaffab47706dd84e57ab3660fac0bc6500f55024aa6" Nov 03 18:00:15.723466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.720371 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:15.723466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.722542 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:00:15.723466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.722594 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-xsc9j" podUID="7b971155-38f4-436e-b6a8-b7224b30286e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:00:15.797531 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.797137 6001 scope.go:122] "RemoveContainer" containerID="8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35" Nov 03 18:00:15.835139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.833859 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\") pod \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " Nov 03 18:00:15.835139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.833967 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\" (UniqueName: \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\") pod \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " Nov 03 18:00:15.835139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.834009 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\") pod \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\" (UID: \"6dc8ff40-f1a3-42fb-b661-fc216e073c85\") " Nov 03 18:00:15.836215 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.836192 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca" pod "6dc8ff40-f1a3-42fb-b661-fc216e073c85" (UID: "6dc8ff40-f1a3-42fb-b661-fc216e073c85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:15.837989 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.837937 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca" pod "6dc8ff40-f1a3-42fb-b661-fc216e073c85" (UID: "6dc8ff40-f1a3-42fb-b661-fc216e073c85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:15.839296 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.839272 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j" pod "6dc8ff40-f1a3-42fb-b661-fc216e073c85" (UID: "6dc8ff40-f1a3-42fb-b661-fc216e073c85"). InnerVolumeSpecName "kube-api-access-d7t8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:15.936186 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.936145 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:15.936186 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.936190 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7t8j\" (UniqueName: \"kubernetes.io/projected/6dc8ff40-f1a3-42fb-b661-fc216e073c85-kube-api-access-d7t8j\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:15.936496 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:15.936208 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc8ff40-f1a3-42fb-b661-fc216e073c85-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:16.329474 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:16.329403 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerStarted","Data":"16552222d6c719eab4d5d995637212537a7420b1f92949dcc0f156800822afc3"} Nov 03 18:00:16.330784 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:16.330752 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" event={"ID":"6dc8ff40-f1a3-42fb-b661-fc216e073c85","Type":"ContainerDied","Data":"e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c"} Nov 03 18:00:16.330784 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:16.330771 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-631ebca0-cv8z9" Nov 03 18:00:16.330983 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:16.330802 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38dcf354b9672b8bc6ed9ed765f16296ca4d8a83add0f802277049b79a0a18c" Nov 03 18:00:17.338922 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:17.338887 6001 generic.go:356] "Generic (PLEG): container finished" podID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerID="16552222d6c719eab4d5d995637212537a7420b1f92949dcc0f156800822afc3" exitCode=60 Nov 03 18:00:17.339468 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:17.338962 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerDied","Data":"16552222d6c719eab4d5d995637212537a7420b1f92949dcc0f156800822afc3"} Nov 03 18:00:17.339468 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:17.339043 6001 scope.go:122] "RemoveContainer" containerID="8248dcc1b558f508dc1797b2fd135a5f92e5caada0fdb0fe23691ff438dd4e35" Nov 03 18:00:17.339586 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:17.339491 6001 scope.go:122] "RemoveContainer" containerID="16552222d6c719eab4d5d995637212537a7420b1f92949dcc0f156800822afc3" Nov 03 18:00:17.339808 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:00:17.339786 6001 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895480-wb572_odh-ai-gateway-infra(7d7b9af2-4bb9-4c92-915b-b18287e97c6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" Nov 03 18:00:17.381456 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:17.381408 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:18.493410 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.493380 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:18.560827 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.560792 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\") pod \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " Nov 03 18:00:18.561049 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.560957 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\" (UniqueName: \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\") pod \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\" (UID: \"7d7b9af2-4bb9-4c92-915b-b18287e97c6d\") " Nov 03 18:00:18.561362 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.561332 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca" pod "7d7b9af2-4bb9-4c92-915b-b18287e97c6d" (UID: "7d7b9af2-4bb9-4c92-915b-b18287e97c6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:18.563208 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.563176 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj" pod "7d7b9af2-4bb9-4c92-915b-b18287e97c6d" (UID: "7d7b9af2-4bb9-4c92-915b-b18287e97c6d"). InnerVolumeSpecName "kube-api-access-f6bxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:18.662662 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.662566 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bxj\" (UniqueName: \"kubernetes.io/projected/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-kube-api-access-f6bxj\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:18.662662 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:18.662600 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d7b9af2-4bb9-4c92-915b-b18287e97c6d-maas-api-service-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:19.354324 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:19.354291 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" Nov 03 18:00:19.354568 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:19.354317 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572" event={"ID":"7d7b9af2-4bb9-4c92-915b-b18287e97c6d","Type":"ContainerDied","Data":"208cf7911e7ba4c4c1cf334d7de16b1cd3f4af5129269f8f466686105ee51603"} Nov 03 18:00:19.354568 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:19.354383 6001 scope.go:122] "RemoveContainer" containerID="16552222d6c719eab4d5d995637212537a7420b1f92949dcc0f156800822afc3" Nov 03 18:00:19.407159 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:19.407125 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:19.419743 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:19.419707 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895480-wb572"] Nov 03 18:00:20.788941 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:20.788907 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" path="/var/lib/kubelet/pods/7d7b9af2-4bb9-4c92-915b-b18287e97c6d/volumes" Nov 03 18:00:22.166394 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.166363 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg"] Nov 03 18:00:22.166973 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.166952 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167095 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.166981 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167095 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167007 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167095 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167016 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167095 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167035 6001 cpu_manager.go:414] "containerMap: removing container" podUID="6dc8ff40-f1a3-42fb-b661-fc216e073c85" containerName="revoke-keys" Nov 03 18:00:22.167095 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167041 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc8ff40-f1a3-42fb-b661-fc216e073c85" containerName="revoke-keys" Nov 03 18:00:22.167398 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167121 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167398 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167137 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc8ff40-f1a3-42fb-b661-fc216e073c85" containerName="revoke-keys" Nov 03 18:00:22.167398 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167150 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.167398 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.167157 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:22.171838 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.171818 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.191129 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.191102 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg"] Nov 03 18:00:22.200835 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.200811 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:22.200957 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.200921 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:00:22.204053 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.204037 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:22.204137 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.204064 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:22.250360 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:00:22.250323 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:00:22.299044 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.299014 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.299209 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.299184 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9kw6\" (UniqueName: \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.299251 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.299212 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.400785 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.400742 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9kw6\" (UniqueName: \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.400785 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.400786 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.401021 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.400833 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.401577 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.401551 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.403185 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.403159 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.425340 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.425272 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9kw6\" (UniqueName: \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\") pod \"maas-api-revoke-sub-8a1652c9-5jrrg\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.484782 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.484736 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:22.789115 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:22.789089 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg"] Nov 03 18:00:22.789252 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:22.789166 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024c47de_c56e_4301_be9f_7b31b641e7c2.slice/crio-46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4 WatchSource:0}: Error finding container 46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4: Status 404 returned error can't find the container with id 46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4 Nov 03 18:00:23.379878 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:23.379769 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" event={"ID":"024c47de-c56e-4301-be9f-7b31b641e7c2","Type":"ContainerStarted","Data":"edf49861f1d2c526b4e596e385848bced9c1b5072f15c0441772b8ded4cdfad9"} Nov 03 18:00:23.379878 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:23.379839 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" event={"ID":"024c47de-c56e-4301-be9f-7b31b641e7c2","Type":"ContainerStarted","Data":"46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4"} Nov 03 18:00:23.424175 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:23.424120 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" podStartSLOduration=1.424105214 podStartE2EDuration="1.424105214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:23.421959372 +0000 UTC m=+6318.755172941" watchObservedRunningTime="2026-11-03 18:00:23.424105214 +0000 UTC m=+6318.757318784" Nov 03 18:00:24.389193 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:24.389097 6001 generic.go:356] "Generic (PLEG): container finished" podID="024c47de-c56e-4301-be9f-7b31b641e7c2" containerID="edf49861f1d2c526b4e596e385848bced9c1b5072f15c0441772b8ded4cdfad9" exitCode=0 Nov 03 18:00:24.389725 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:24.389198 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" event={"ID":"024c47de-c56e-4301-be9f-7b31b641e7c2","Type":"ContainerDied","Data":"edf49861f1d2c526b4e596e385848bced9c1b5072f15c0441772b8ded4cdfad9"} Nov 03 18:00:25.839275 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.839244 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:25.955478 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.943756 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\" (UniqueName: \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\") pod \"024c47de-c56e-4301-be9f-7b31b641e7c2\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " Nov 03 18:00:25.955478 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.943880 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\") pod \"024c47de-c56e-4301-be9f-7b31b641e7c2\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " Nov 03 18:00:25.955478 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.943954 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\") pod \"024c47de-c56e-4301-be9f-7b31b641e7c2\" (UID: \"024c47de-c56e-4301-be9f-7b31b641e7c2\") " Nov 03 18:00:25.956879 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.956415 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca" pod "024c47de-c56e-4301-be9f-7b31b641e7c2" (UID: "024c47de-c56e-4301-be9f-7b31b641e7c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:25.975139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.975095 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6" pod "024c47de-c56e-4301-be9f-7b31b641e7c2" (UID: "024c47de-c56e-4301-be9f-7b31b641e7c2"). InnerVolumeSpecName "kube-api-access-h9kw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:25.975826 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:25.975767 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca" pod "024c47de-c56e-4301-be9f-7b31b641e7c2" (UID: "024c47de-c56e-4301-be9f-7b31b641e7c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:26.070542 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.063512 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:26.070542 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.063564 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9kw6\" (UniqueName: \"kubernetes.io/projected/024c47de-c56e-4301-be9f-7b31b641e7c2-kube-api-access-h9kw6\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:26.070542 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.063582 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024c47de-c56e-4301-be9f-7b31b641e7c2-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:26.418869 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.418781 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" event={"ID":"024c47de-c56e-4301-be9f-7b31b641e7c2","Type":"ContainerDied","Data":"46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4"} Nov 03 18:00:26.418869 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.418796 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a1652c9-5jrrg" Nov 03 18:00:26.418869 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:26.418831 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46a198cc3610139a30eb2c0d2413e524d9d2d292363dc5be98490ab71ceeafc4" Nov 03 18:00:32.603424 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603385 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m"] Nov 03 18:00:32.603902 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603860 6001 cpu_manager.go:414] "containerMap: removing container" podUID="024c47de-c56e-4301-be9f-7b31b641e7c2" containerName="revoke-keys" Nov 03 18:00:32.603902 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603873 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024c47de-c56e-4301-be9f-7b31b641e7c2" containerName="revoke-keys" Nov 03 18:00:32.603902 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603895 6001 cpu_manager.go:414] "containerMap: removing container" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:32.603902 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603901 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d7b9af2-4bb9-4c92-915b-b18287e97c6d" containerName="cleanup" Nov 03 18:00:32.604079 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.603984 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="024c47de-c56e-4301-be9f-7b31b641e7c2" containerName="revoke-keys" Nov 03 18:00:32.607228 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.607201 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.634858 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.634821 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:00:32.635089 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.634863 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:32.635089 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.634975 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:32.637531 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.637485 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m"] Nov 03 18:00:32.722621 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.722588 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:32.739881 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.739841 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.740075 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.739894 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwvvh\" (UniqueName: \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.740075 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.740021 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.841621 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.841587 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.841819 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.841635 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwvvh\" (UniqueName: \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.841819 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.841676 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.842404 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.842377 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.844159 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.844134 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.866194 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.866116 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwvvh\" (UniqueName: \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\") pod \"maas-api-revoke-sub-f6fa6b79-lwx7m\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:32.922312 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:32.922264 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:33.229687 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:33.229656 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m"] Nov 03 18:00:33.230618 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:33.230589 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84559ce5_8209_44d7_8424_9132351db48d.slice/crio-5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d WatchSource:0}: Error finding container 5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d: Status 404 returned error can't find the container with id 5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d Nov 03 18:00:33.461741 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:33.461707 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" event={"ID":"84559ce5-8209-44d7-8424-9132351db48d","Type":"ContainerStarted","Data":"5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d"} Nov 03 18:00:34.471675 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:34.471573 6001 generic.go:356] "Generic (PLEG): container finished" podID="84559ce5-8209-44d7-8424-9132351db48d" containerID="3d31a4f749dbfcf2f75734bbebe35a10c6529d6c2fcd1b532a66bad729ab7357" exitCode=0 Nov 03 18:00:34.471675 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:34.471656 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" event={"ID":"84559ce5-8209-44d7-8424-9132351db48d","Type":"ContainerDied","Data":"3d31a4f749dbfcf2f75734bbebe35a10c6529d6c2fcd1b532a66bad729ab7357"} Nov 03 18:00:35.856910 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:35.856871 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:36.027469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.007955 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\") pod \"84559ce5-8209-44d7-8424-9132351db48d\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " Nov 03 18:00:36.027469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.027251 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\") pod \"84559ce5-8209-44d7-8424-9132351db48d\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " Nov 03 18:00:36.027469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.027297 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\" (UniqueName: \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\") pod \"84559ce5-8209-44d7-8424-9132351db48d\" (UID: \"84559ce5-8209-44d7-8424-9132351db48d\") " Nov 03 18:00:36.028469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.028269 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca" pod "84559ce5-8209-44d7-8424-9132351db48d" (UID: "84559ce5-8209-44d7-8424-9132351db48d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:36.030652 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.030606 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh" pod "84559ce5-8209-44d7-8424-9132351db48d" (UID: "84559ce5-8209-44d7-8424-9132351db48d"). InnerVolumeSpecName "kube-api-access-zwvvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:36.030777 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.030714 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca" pod "84559ce5-8209-44d7-8424-9132351db48d" (UID: "84559ce5-8209-44d7-8424-9132351db48d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:36.128740 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.128662 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84559ce5-8209-44d7-8424-9132351db48d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:36.128740 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.128689 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84559ce5-8209-44d7-8424-9132351db48d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:36.128740 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.128699 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwvvh\" (UniqueName: \"kubernetes.io/projected/84559ce5-8209-44d7-8424-9132351db48d-kube-api-access-zwvvh\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:36.511469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.510516 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" event={"ID":"84559ce5-8209-44d7-8424-9132351db48d","Type":"ContainerDied","Data":"5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d"} Nov 03 18:00:36.511469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.510668 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d7018c652c4f6d042b7aef399587bce20cf66160b5ee1adb3bdc53392fbfc1d" Nov 03 18:00:36.511469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:36.510795 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6fa6b79-lwx7m" Nov 03 18:00:42.957202 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.957164 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq"] Nov 03 18:00:42.957652 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.957588 6001 cpu_manager.go:414] "containerMap: removing container" podUID="84559ce5-8209-44d7-8424-9132351db48d" containerName="revoke-keys" Nov 03 18:00:42.957652 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.957600 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84559ce5-8209-44d7-8424-9132351db48d" containerName="revoke-keys" Nov 03 18:00:42.957748 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.957705 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="84559ce5-8209-44d7-8424-9132351db48d" containerName="revoke-keys" Nov 03 18:00:42.960889 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.960870 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:42.990732 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.990696 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:42.990732 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.990707 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:00:42.994612 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.994579 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:42.994612 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.994611 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:42.994807 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:42.994770 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq"] Nov 03 18:00:43.002334 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.002306 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tdx8\" (UniqueName: \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.002526 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.002508 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.002581 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.002561 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.104060 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.104026 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.104253 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.104075 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.104253 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.104123 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tdx8\" (UniqueName: \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.104741 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.104706 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.106542 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.106522 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.128245 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.128218 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tdx8\" (UniqueName: \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\") pod \"maas-api-revoke-sub-97f54be0-b7psq\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.275320 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.275281 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:43.579627 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:43.579590 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1846c659_36d7_466a_87a4_e87cfc024778.slice/crio-9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6 WatchSource:0}: Error finding container 9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6: Status 404 returned error can't find the container with id 9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6 Nov 03 18:00:43.579973 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:43.579948 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq"] Nov 03 18:00:44.562931 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:44.562890 6001 generic.go:356] "Generic (PLEG): container finished" podID="1846c659-36d7-466a-87a4-e87cfc024778" containerID="443de617aa6ed504da13a11532872b7512d745cda08c0828cbcb483b5cfa44fc" exitCode=0 Nov 03 18:00:44.563351 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:44.562942 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" event={"ID":"1846c659-36d7-466a-87a4-e87cfc024778","Type":"ContainerDied","Data":"443de617aa6ed504da13a11532872b7512d745cda08c0828cbcb483b5cfa44fc"} Nov 03 18:00:44.563351 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:44.563014 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" event={"ID":"1846c659-36d7-466a-87a4-e87cfc024778","Type":"ContainerStarted","Data":"9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6"} Nov 03 18:00:46.016872 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.016836 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:46.035537 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.035499 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\" (UniqueName: \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\") pod \"1846c659-36d7-466a-87a4-e87cfc024778\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " Nov 03 18:00:46.035774 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.035568 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\") pod \"1846c659-36d7-466a-87a4-e87cfc024778\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " Nov 03 18:00:46.035774 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.035631 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\") pod \"1846c659-36d7-466a-87a4-e87cfc024778\" (UID: \"1846c659-36d7-466a-87a4-e87cfc024778\") " Nov 03 18:00:46.036092 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.036063 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca" pod "1846c659-36d7-466a-87a4-e87cfc024778" (UID: "1846c659-36d7-466a-87a4-e87cfc024778"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:46.037899 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.037873 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8" pod "1846c659-36d7-466a-87a4-e87cfc024778" (UID: "1846c659-36d7-466a-87a4-e87cfc024778"). InnerVolumeSpecName "kube-api-access-9tdx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:46.038006 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.037908 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca" pod "1846c659-36d7-466a-87a4-e87cfc024778" (UID: "1846c659-36d7-466a-87a4-e87cfc024778"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:46.137108 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.137063 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:46.137108 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.137097 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tdx8\" (UniqueName: \"kubernetes.io/projected/1846c659-36d7-466a-87a4-e87cfc024778-kube-api-access-9tdx8\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:46.137108 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.137114 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1846c659-36d7-466a-87a4-e87cfc024778-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:46.619483 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.619269 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" event={"ID":"1846c659-36d7-466a-87a4-e87cfc024778","Type":"ContainerDied","Data":"9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6"} Nov 03 18:00:46.619483 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.619374 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9693c38bac02824cdc23f7dae35c59a260191dd66f5e2a75d37922976f4623b6" Nov 03 18:00:46.619483 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.619471 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f54be0-b7psq" Nov 03 18:00:46.968680 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.968570 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:00:46.969048 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:46.969012 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-fmxr5" podUID="c9732c01-e962-45c8-a46c-c83c88a65a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:00:52.251052 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:00:52.251004 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:00:53.312146 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.312110 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k"] Nov 03 18:00:53.312604 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.312588 6001 cpu_manager.go:414] "containerMap: removing container" podUID="1846c659-36d7-466a-87a4-e87cfc024778" containerName="revoke-keys" Nov 03 18:00:53.312604 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.312601 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1846c659-36d7-466a-87a4-e87cfc024778" containerName="revoke-keys" Nov 03 18:00:53.312714 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.312703 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="1846c659-36d7-466a-87a4-e87cfc024778" containerName="revoke-keys" Nov 03 18:00:53.316125 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.316107 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.334682 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.333609 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:00:53.346179 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.346144 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:00:53.346179 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.346150 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:00:53.346425 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.346238 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:00:53.348829 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.348802 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k"] Nov 03 18:00:53.426173 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.426138 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.426173 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.426178 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzn5\" (UniqueName: \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.426418 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.426200 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.527471 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.527395 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.527680 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.527482 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzn5\" (UniqueName: \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.527680 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.527514 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.528120 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.528099 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.530023 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.529998 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.551731 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.551701 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzn5\" (UniqueName: \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\") pod \"maas-api-revoke-sub-027be112-zml2k\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.636894 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.636792 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:53.943671 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:53.943635 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k"] Nov 03 18:00:53.944225 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:00:53.944193 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e789dd_8f0f_48e2_82c8_aa57890c19b6.slice/crio-356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa WatchSource:0}: Error finding container 356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa: Status 404 returned error can't find the container with id 356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa Nov 03 18:00:54.679537 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:54.678849 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" event={"ID":"a0e789dd-8f0f-48e2-82c8-aa57890c19b6","Type":"ContainerStarted","Data":"31c57ee9118db2ac8f3474dc4b47710203dd2c8b2480f5958841d7f71f773458"} Nov 03 18:00:54.680086 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:54.680055 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" event={"ID":"a0e789dd-8f0f-48e2-82c8-aa57890c19b6","Type":"ContainerStarted","Data":"356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa"} Nov 03 18:00:55.695469 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:55.695417 6001 generic.go:356] "Generic (PLEG): container finished" podID="a0e789dd-8f0f-48e2-82c8-aa57890c19b6" containerID="31c57ee9118db2ac8f3474dc4b47710203dd2c8b2480f5958841d7f71f773458" exitCode=0 Nov 03 18:00:55.696150 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:55.696128 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" event={"ID":"a0e789dd-8f0f-48e2-82c8-aa57890c19b6","Type":"ContainerDied","Data":"31c57ee9118db2ac8f3474dc4b47710203dd2c8b2480f5958841d7f71f773458"} Nov 03 18:00:57.123505 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.123271 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:57.136721 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.136689 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\") pod \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " Nov 03 18:00:57.136926 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.136795 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\" (UniqueName: \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\") pod \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " Nov 03 18:00:57.136926 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.136847 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\") pod \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\" (UID: \"a0e789dd-8f0f-48e2-82c8-aa57890c19b6\") " Nov 03 18:00:57.137142 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.137115 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca" pod "a0e789dd-8f0f-48e2-82c8-aa57890c19b6" (UID: "a0e789dd-8f0f-48e2-82c8-aa57890c19b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:57.138926 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.138899 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5" pod "a0e789dd-8f0f-48e2-82c8-aa57890c19b6" (UID: "a0e789dd-8f0f-48e2-82c8-aa57890c19b6"). InnerVolumeSpecName "kube-api-access-4wzn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:57.139083 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.138998 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca" pod "a0e789dd-8f0f-48e2-82c8-aa57890c19b6" (UID: "a0e789dd-8f0f-48e2-82c8-aa57890c19b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:00:57.240179 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.239974 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzn5\" (UniqueName: \"kubernetes.io/projected/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-kube-api-access-4wzn5\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:57.240179 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.240008 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:57.240179 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.240019 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0e789dd-8f0f-48e2-82c8-aa57890c19b6-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:00:57.720361 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.720335 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" Nov 03 18:00:57.720681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.720331 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-027be112-zml2k" event={"ID":"a0e789dd-8f0f-48e2-82c8-aa57890c19b6","Type":"ContainerDied","Data":"356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa"} Nov 03 18:00:57.720681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:00:57.720476 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="356ae2ee09f5b508a84448f7c7279c3bd22b6838c53ec68ee3f83838d438c1aa" Nov 03 18:01:03.652287 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.652246 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk"] Nov 03 18:01:03.652758 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.652687 6001 cpu_manager.go:414] "containerMap: removing container" podUID="a0e789dd-8f0f-48e2-82c8-aa57890c19b6" containerName="revoke-keys" Nov 03 18:01:03.652758 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.652699 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e789dd-8f0f-48e2-82c8-aa57890c19b6" containerName="revoke-keys" Nov 03 18:01:03.652852 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.652803 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e789dd-8f0f-48e2-82c8-aa57890c19b6" containerName="revoke-keys" Nov 03 18:01:03.657115 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.657085 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.675417 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.674709 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:01:03.686736 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.686705 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:01:03.689630 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.689598 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk"] Nov 03 18:01:03.698578 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.698539 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqkc\" (UniqueName: \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.698752 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.698696 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.699020 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.698983 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.701282 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.701257 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:01:03.701418 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.701364 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:01:03.800211 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.800165 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqkc\" (UniqueName: \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.800436 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.800216 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.800436 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.800293 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.801018 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.800996 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.802704 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.802681 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.824790 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.824755 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqkc\" (UniqueName: \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\") pod \"maas-api-revoke-sub-b486bbc9-7t6kk\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:03.971196 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:03.971107 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:04.275387 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:04.275353 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk"] Nov 03 18:01:04.275780 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:01:04.275753 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15241b2d_9413_4e23_9b64_2ff7763dc7c7.slice/crio-31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783 WatchSource:0}: Error finding container 31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783: Status 404 returned error can't find the container with id 31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783 Nov 03 18:01:04.772026 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:04.771985 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" event={"ID":"15241b2d-9413-4e23-9b64-2ff7763dc7c7","Type":"ContainerStarted","Data":"16d73fa67d9e212bbfe4d26e1276d5dc5bb496a9d8b11fe3107d4d8015a61faa"} Nov 03 18:01:04.772550 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:04.772043 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" event={"ID":"15241b2d-9413-4e23-9b64-2ff7763dc7c7","Type":"ContainerStarted","Data":"31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783"} Nov 03 18:01:04.815483 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:04.815365 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" podStartSLOduration=1.8153529160000001 podStartE2EDuration="1.815352916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:04.81454696 +0000 UTC m=+6360.147760529" watchObservedRunningTime="2026-11-03 18:01:04.815352916 +0000 UTC m=+6360.148566530" Nov 03 18:01:05.782543 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:05.782508 6001 generic.go:356] "Generic (PLEG): container finished" podID="15241b2d-9413-4e23-9b64-2ff7763dc7c7" containerID="16d73fa67d9e212bbfe4d26e1276d5dc5bb496a9d8b11fe3107d4d8015a61faa" exitCode=0 Nov 03 18:01:05.783120 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:05.782600 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" event={"ID":"15241b2d-9413-4e23-9b64-2ff7763dc7c7","Type":"ContainerDied","Data":"16d73fa67d9e212bbfe4d26e1276d5dc5bb496a9d8b11fe3107d4d8015a61faa"} Nov 03 18:01:07.228500 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.228220 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:07.237340 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.237312 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\") pod \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " Nov 03 18:01:07.237519 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.237387 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\") pod \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " Nov 03 18:01:07.237519 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.237429 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\" (UniqueName: \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\") pod \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\" (UID: \"15241b2d-9413-4e23-9b64-2ff7763dc7c7\") " Nov 03 18:01:07.237928 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.237899 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca" pod "15241b2d-9413-4e23-9b64-2ff7763dc7c7" (UID: "15241b2d-9413-4e23-9b64-2ff7763dc7c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:07.239833 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.239807 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca" pod "15241b2d-9413-4e23-9b64-2ff7763dc7c7" (UID: "15241b2d-9413-4e23-9b64-2ff7763dc7c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:07.239913 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.239854 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc" pod "15241b2d-9413-4e23-9b64-2ff7763dc7c7" (UID: "15241b2d-9413-4e23-9b64-2ff7763dc7c7"). InnerVolumeSpecName "kube-api-access-8zqkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:07.339051 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.339018 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:07.339051 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.339049 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15241b2d-9413-4e23-9b64-2ff7763dc7c7-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:07.339051 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.339058 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqkc\" (UniqueName: \"kubernetes.io/projected/15241b2d-9413-4e23-9b64-2ff7763dc7c7-kube-api-access-8zqkc\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:07.829020 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.828985 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" Nov 03 18:01:07.829229 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.829019 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b486bbc9-7t6kk" event={"ID":"15241b2d-9413-4e23-9b64-2ff7763dc7c7","Type":"ContainerDied","Data":"31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783"} Nov 03 18:01:07.829229 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:07.829080 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c9aee1ac0ff48ef972553ab014bbb840bd1ec2775b9d6df8be84f20067f783" Nov 03 18:01:14.087338 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.087292 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84"] Nov 03 18:01:14.087860 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.087740 6001 cpu_manager.go:414] "containerMap: removing container" podUID="15241b2d-9413-4e23-9b64-2ff7763dc7c7" containerName="revoke-keys" Nov 03 18:01:14.087860 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.087753 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15241b2d-9413-4e23-9b64-2ff7763dc7c7" containerName="revoke-keys" Nov 03 18:01:14.087860 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.087815 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="15241b2d-9413-4e23-9b64-2ff7763dc7c7" containerName="revoke-keys" Nov 03 18:01:14.091958 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.091936 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.122040 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.122014 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:01:14.122040 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.122024 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:01:14.122241 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.122047 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:01:14.122241 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.122100 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:01:14.124727 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.124708 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84"] Nov 03 18:01:14.204800 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.204761 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.204998 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.204896 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.204998 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.204960 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b67p\" (UniqueName: \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.305915 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.305877 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.306112 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.305948 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b67p\" (UniqueName: \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.306112 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.306019 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.306787 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.306766 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.308419 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.308397 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.330231 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.330191 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b67p\" (UniqueName: \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\") pod \"maas-api-revoke-sub-0a28e5a4-c9d84\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.406104 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.406006 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:14.709834 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.709796 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84"] Nov 03 18:01:14.710806 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:01:14.710776 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd918a17e_e73c_42ff_ac21_228d3327f278.slice/crio-6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598 WatchSource:0}: Error finding container 6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598: Status 404 returned error can't find the container with id 6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598 Nov 03 18:01:14.873875 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:14.873837 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" event={"ID":"d918a17e-e73c-42ff-ac21-228d3327f278","Type":"ContainerStarted","Data":"6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598"} Nov 03 18:01:15.242080 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:15.242035 6001 scope.go:122] "RemoveContainer" containerID="cf3ab03498d1dbcedcf04571c4d6aa9acb65bab8ab7a1952d35323b04a2c9012" Nov 03 18:01:15.253209 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:15.253187 6001 scope.go:122] "RemoveContainer" containerID="2979c7fbc015e47ae919839ed7d6db610ea6815bcf8585e81aca42d1d3e7ecac" Nov 03 18:01:15.882874 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:15.882828 6001 generic.go:356] "Generic (PLEG): container finished" podID="d918a17e-e73c-42ff-ac21-228d3327f278" containerID="bab54c3f9899f6e142cb6125889cd883600e2e44fa2ca1890b466a9411eb6e87" exitCode=0 Nov 03 18:01:15.883107 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:15.882886 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" event={"ID":"d918a17e-e73c-42ff-ac21-228d3327f278","Type":"ContainerDied","Data":"bab54c3f9899f6e142cb6125889cd883600e2e44fa2ca1890b466a9411eb6e87"} Nov 03 18:01:17.294711 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.294673 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:17.340649 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.340604 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\" (UniqueName: \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\") pod \"d918a17e-e73c-42ff-ac21-228d3327f278\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " Nov 03 18:01:17.340896 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.340686 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\") pod \"d918a17e-e73c-42ff-ac21-228d3327f278\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " Nov 03 18:01:17.340896 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.340836 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\") pod \"d918a17e-e73c-42ff-ac21-228d3327f278\" (UID: \"d918a17e-e73c-42ff-ac21-228d3327f278\") " Nov 03 18:01:17.341142 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.341110 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca" pod "d918a17e-e73c-42ff-ac21-228d3327f278" (UID: "d918a17e-e73c-42ff-ac21-228d3327f278"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:17.370468 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.364812 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p" pod "d918a17e-e73c-42ff-ac21-228d3327f278" (UID: "d918a17e-e73c-42ff-ac21-228d3327f278"). InnerVolumeSpecName "kube-api-access-9b67p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:17.370468 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.364910 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca" pod "d918a17e-e73c-42ff-ac21-228d3327f278" (UID: "d918a17e-e73c-42ff-ac21-228d3327f278"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:17.441726 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.441679 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b67p\" (UniqueName: \"kubernetes.io/projected/d918a17e-e73c-42ff-ac21-228d3327f278-kube-api-access-9b67p\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:17.441726 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.441720 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:17.441726 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.441730 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d918a17e-e73c-42ff-ac21-228d3327f278-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:17.919470 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.917325 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" event={"ID":"d918a17e-e73c-42ff-ac21-228d3327f278","Type":"ContainerDied","Data":"6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598"} Nov 03 18:01:17.919470 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.917398 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6349b62de430dd2b214e3aa9989e94594d2e78dd2a8f63205b9abcdb47511598" Nov 03 18:01:17.919470 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:17.917470 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a28e5a4-c9d84" Nov 03 18:01:18.007467 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:18.006570 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:01:18.007467 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:18.006663 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:01:22.250766 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:01:22.250729 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:01:24.432730 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.432650 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p"] Nov 03 18:01:24.433146 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.433131 6001 cpu_manager.go:414] "containerMap: removing container" podUID="d918a17e-e73c-42ff-ac21-228d3327f278" containerName="revoke-keys" Nov 03 18:01:24.433202 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.433149 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d918a17e-e73c-42ff-ac21-228d3327f278" containerName="revoke-keys" Nov 03 18:01:24.433245 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.433229 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="d918a17e-e73c-42ff-ac21-228d3327f278" containerName="revoke-keys" Nov 03 18:01:24.436884 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.436857 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.454002 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.453942 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:01:24.466602 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.466570 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:01:24.469166 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.469140 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p"] Nov 03 18:01:24.473796 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.473764 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:01:24.473938 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.473855 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:01:24.532520 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.532476 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.532704 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.532547 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.532704 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.532677 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4n5k\" (UniqueName: \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.633689 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.633641 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.633895 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.633714 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.633895 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.633760 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4n5k\" (UniqueName: \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.634332 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.634304 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.636178 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.636153 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.657920 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.657884 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4n5k\" (UniqueName: \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\") pod \"maas-api-revoke-sub-86ed6347-l6z6p\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:24.750345 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:24.750304 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:25.055602 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:25.055560 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p"] Nov 03 18:01:25.056101 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:01:25.056071 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d672c99_eaf6_44b0_a6a7_c3a4107b9b7d.slice/crio-4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7 WatchSource:0}: Error finding container 4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7: Status 404 returned error can't find the container with id 4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7 Nov 03 18:01:25.967222 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:25.967145 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" event={"ID":"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d","Type":"ContainerStarted","Data":"5ba79172bc974bed2602baf30b133baf3fa588c80e7091e10ee173fbb16077e2"} Nov 03 18:01:25.967709 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:25.967219 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" event={"ID":"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d","Type":"ContainerStarted","Data":"4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7"} Nov 03 18:01:26.012123 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:26.012055 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" podStartSLOduration=2.012037294 podStartE2EDuration="2.012037294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:26.010302136 +0000 UTC m=+6381.343515706" watchObservedRunningTime="2026-11-03 18:01:26.012037294 +0000 UTC m=+6381.345250864" Nov 03 18:01:27.010999 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:27.010960 6001 generic.go:356] "Generic (PLEG): container finished" podID="3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" containerID="5ba79172bc974bed2602baf30b133baf3fa588c80e7091e10ee173fbb16077e2" exitCode=0 Nov 03 18:01:27.011640 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:27.011355 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" event={"ID":"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d","Type":"ContainerDied","Data":"5ba79172bc974bed2602baf30b133baf3fa588c80e7091e10ee173fbb16077e2"} Nov 03 18:01:28.353623 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.353581 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:28.518413 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.518357 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\") pod \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " Nov 03 18:01:28.518620 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.518570 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\" (UniqueName: \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\") pod \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " Nov 03 18:01:28.518681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.518662 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\") pod \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\" (UID: \"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d\") " Nov 03 18:01:28.518972 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.518945 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca" pod "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" (UID: "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:28.519100 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.519084 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:28.520560 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.520535 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca" pod "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" (UID: "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:28.520709 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.520688 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k" pod "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" (UID: "3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d"). InnerVolumeSpecName "kube-api-access-m4n5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:28.620714 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.620637 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4n5k\" (UniqueName: \"kubernetes.io/projected/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-kube-api-access-m4n5k\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:28.620714 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:28.620670 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:29.029221 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:29.029194 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" Nov 03 18:01:29.029405 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:29.029222 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86ed6347-l6z6p" event={"ID":"3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d","Type":"ContainerDied","Data":"4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7"} Nov 03 18:01:29.029405 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:29.029267 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dbb46401dbe8c09b71a835aae630bbd1355bb65a0ee2351ba9ce39581b3ddf7" Nov 03 18:01:32.036708 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:32.036667 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg"] Nov 03 18:01:32.049208 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:32.049178 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-462bg"] Nov 03 18:01:32.787935 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:32.787902 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb85b285-068e-458f-8286-77dc14b565aa" path="/var/lib/kubelet/pods/cb85b285-068e-458f-8286-77dc14b565aa/volumes" Nov 03 18:01:34.787925 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.787895 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q"] Nov 03 18:01:34.788383 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.788307 6001 cpu_manager.go:414] "containerMap: removing container" podUID="3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" containerName="revoke-keys" Nov 03 18:01:34.788383 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.788319 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" containerName="revoke-keys" Nov 03 18:01:34.788496 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.788393 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d" containerName="revoke-keys" Nov 03 18:01:34.791755 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.791736 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.810110 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.810050 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q"] Nov 03 18:01:34.817125 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.817091 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:01:34.817298 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.817284 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-q4q9f\"" Nov 03 18:01:34.829688 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.829660 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 18:01:34.829827 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.829762 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 18:01:34.886609 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.886560 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.886839 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.886719 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.886839 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.886754 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttf7\" (UniqueName: \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.987434 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.987385 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.987671 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.987538 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.987671 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.987572 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttf7\" (UniqueName: \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.988244 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.988217 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:34.989948 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:34.989922 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:35.011931 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:35.011902 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttf7\" (UniqueName: \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\") pod \"maas-api-revoke-sub-ef750943-fjq5q\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:35.105259 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:35.105160 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:35.411556 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:35.411521 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q"] Nov 03 18:01:35.411885 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:01:35.411860 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd38b39_7f12_4eb0_87fc_be492ef9a0ee.slice/crio-a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84 WatchSource:0}: Error finding container a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84: Status 404 returned error can't find the container with id a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84 Nov 03 18:01:36.076473 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:36.076413 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" event={"ID":"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee","Type":"ContainerStarted","Data":"87040b8c4fbcd9c5a2791624cca4d90104562e25a283cd917bd057b0d41597cf"} Nov 03 18:01:36.076911 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:36.076488 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" event={"ID":"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee","Type":"ContainerStarted","Data":"a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84"} Nov 03 18:01:36.120520 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:36.120466 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" podStartSLOduration=2.120437004 podStartE2EDuration="2.120437004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:36.118885577 +0000 UTC m=+6391.452099146" watchObservedRunningTime="2026-11-03 18:01:36.120437004 +0000 UTC m=+6391.453650573" Nov 03 18:01:37.123043 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:37.122459 6001 generic.go:356] "Generic (PLEG): container finished" podID="fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" containerID="87040b8c4fbcd9c5a2791624cca4d90104562e25a283cd917bd057b0d41597cf" exitCode=0 Nov 03 18:01:37.123043 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:37.122568 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" event={"ID":"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee","Type":"ContainerDied","Data":"87040b8c4fbcd9c5a2791624cca4d90104562e25a283cd917bd057b0d41597cf"} Nov 03 18:01:38.270571 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.270544 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:38.346075 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.346043 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\") pod \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " Nov 03 18:01:38.346262 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.346083 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\") pod \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " Nov 03 18:01:38.346262 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.346110 6001 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\" (UniqueName: \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\") pod \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\" (UID: \"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee\") " Nov 03 18:01:38.346574 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.346547 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca" pod "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" (UID: "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:38.348342 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.348318 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca" pod "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" (UID: "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:38.348472 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.348398 6001 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7" pod "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" (UID: "fbd38b39-7f12-4eb0-87fc-be492ef9a0ee"). InnerVolumeSpecName "kube-api-access-xttf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:38.447856 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.447761 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-xks-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:38.447856 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.447797 6001 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-maas-api-openshift-ca\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:38.447856 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:38.447810 6001 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttf7\" (UniqueName: \"kubernetes.io/projected/fbd38b39-7f12-4eb0-87fc-be492ef9a0ee-kube-api-access-xttf7\") on node \"ip-10-0-129-130.ec2.internal\" DevicePath \"\"" Nov 03 18:01:39.135475 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:39.135370 6001 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" Nov 03 18:01:39.135475 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:39.135399 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef750943-fjq5q" event={"ID":"fbd38b39-7f12-4eb0-87fc-be492ef9a0ee","Type":"ContainerDied","Data":"a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84"} Nov 03 18:01:39.135475 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:39.135471 6001 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9aadcfc870128310539f674163e4fced933d1240d66440a32e8aa31265ecd84" Nov 03 18:01:48.007534 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:48.003618 6001 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-bgct8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:01:48.007534 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:48.003728 6001 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-bgct8" podUID="90e58ef7-2fe2-4060-af83-3d06ba7cbea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:01:48.029353 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:48.029317 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79"] Nov 03 18:01:48.036776 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:48.036741 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-7bc79"] Nov 03 18:01:48.788272 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:48.788240 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd241444-9fa1-4750-9cd5-23ea00f38d71" path="/var/lib/kubelet/pods/cd241444-9fa1-4750-9cd5-23ea00f38d71/volumes" Nov 03 18:01:52.254085 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:01:52.254043 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:01:57.194985 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:01:57.194942 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4c8d89d8-khtcc_7ac3d41f-22e1-4ef3-8389-4fc3e73d5196/authorino/0.log" Nov 03 18:02:11.516731 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:11.516698 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-01948d0b-464q2_6566161b-4221-4cb2-b878-10ba73c722f8/revoke-keys/0.log" Nov 03 18:02:11.685323 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:11.685290 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-027be112-zml2k_a0e789dd-8f0f-48e2-82c8-aa57890c19b6/revoke-keys/0.log" Nov 03 18:02:11.855132 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:11.855006 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0a28e5a4-c9d84_d918a17e-e73c-42ff-ac21-228d3327f278/revoke-keys/0.log" Nov 03 18:02:12.032226 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.032195 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-55d5ae6a-bdn76_c93f4803-6fa1-4108-96a4-c2ea1faa7609/revoke-keys/0.log" Nov 03 18:02:12.238415 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.238381 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-631ebca0-cv8z9_6dc8ff40-f1a3-42fb-b661-fc216e073c85/revoke-keys/0.log" Nov 03 18:02:12.418828 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.418798 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-69c09e8b-wqv8n_4c8f4aef-6fb9-4cd3-b7d8-ce0c2e63d3df/revoke-keys/0.log" Nov 03 18:02:12.604148 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.604069 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6c9144af-79pjm_d05ee6e6-7abb-406a-bee8-713c61afe7b1/revoke-keys/0.log" Nov 03 18:02:12.791872 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.791837 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-71a86c5d-7tghv_1e5d4562-808a-4c15-875d-683e2b5cd59c/revoke-keys/0.log" Nov 03 18:02:12.983700 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:12.983667 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84d39b4f-9x644_fe6cae8c-3e17-441a-80fc-07c04dee42e8/revoke-keys/0.log" Nov 03 18:02:13.167601 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:13.167572 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-86ed6347-l6z6p_3d672c99-eaf6-44b0-a6a7-c3a4107b9b7d/revoke-keys/0.log" Nov 03 18:02:13.347770 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:13.347679 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8a1652c9-5jrrg_024c47de-c56e-4301-be9f-7b31b641e7c2/revoke-keys/0.log" Nov 03 18:02:13.534484 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:13.534430 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-97f54be0-b7psq_1846c659-36d7-466a-87a4-e87cfc024778/revoke-keys/0.log" Nov 03 18:02:13.716734 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:13.716650 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-990be415-6xhvf_e3349b5e-c4a1-4e9a-84a8-7a4b0f8ca678/revoke-keys/0.log" Nov 03 18:02:13.914345 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:13.914316 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b486bbc9-7t6kk_15241b2d-9413-4e23-9b64-2ff7763dc7c7/revoke-keys/0.log" Nov 03 18:02:14.108948 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:14.108919 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cb4bbb40-vshjt_137500b0-b3aa-43a8-9cef-88ca35eb15b1/revoke-keys/0.log" Nov 03 18:02:14.287261 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:14.287233 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e71c6725-8cm8f_bb327586-8632-4578-8efe-cd725ba5ee93/revoke-keys/0.log" Nov 03 18:02:14.473182 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:14.473159 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ef750943-fjq5q_fbd38b39-7f12-4eb0-87fc-be492ef9a0ee/revoke-keys/0.log" Nov 03 18:02:14.653003 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:14.652971 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f6d6c53d-hpzgq_404cb8bf-0288-4077-b53a-5c72533b41d1/revoke-keys/0.log" Nov 03 18:02:14.833371 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:14.833290 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f6fa6b79-lwx7m_84559ce5-8209-44d7-8424-9132351db48d/revoke-keys/0.log" Nov 03 18:02:16.862470 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:16.862426 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4c8d89d8-khtcc_7ac3d41f-22e1-4ef3-8389-4fc3e73d5196/authorino/0.log" Nov 03 18:02:17.458827 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:17.458798 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zd2c8_cb8bbc2c-4a30-45a4-a981-63996002b0c4/kuadrant-console-plugin/0.log" Nov 03 18:02:18.029429 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:18.029385 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f"] Nov 03 18:02:18.041748 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:18.041717 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e71c6725-8cm8f"] Nov 03 18:02:18.788465 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:18.788416 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb327586-8632-4578-8efe-cd725ba5ee93" path="/var/lib/kubelet/pods/bb327586-8632-4578-8efe-cd725ba5ee93/volumes" Nov 03 18:02:19.593594 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:19.593566 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-kgglr_601ebbab-626e-4679-a41e-437cdf951a1c/kube-auth-proxy/0.log" Nov 03 18:02:20.398024 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:20.397990 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-bgct8_90e58ef7-2fe2-4060-af83-3d06ba7cbea9/istio-proxy/0.log" Nov 03 18:02:20.822962 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:20.822921 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fmxr5_c9732c01-e962-45c8-a46c-c83c88a65a4e/istio-proxy/0.log" Nov 03 18:02:21.409069 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:21.409034 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xsc9j_7b971155-38f4-436e-b6a8-b7224b30286e/istio-proxy/0.log" Nov 03 18:02:22.209890 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:22.209847 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d85d667ff-kc82s_9139f8c8-371a-44d5-8d0e-636f0eb11e48/payload-processing/0.log" Nov 03 18:02:22.250269 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:02:22.250237 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:02:23.631966 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:23.631931 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-gf9p8_e52d3d63-9c13-44bd-beca-7370089217c4/main/0.log" Nov 03 18:02:23.992488 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:23.992454 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-sjb24_43be8571-61d3-47ea-aeb5-67d131720b19/main/0.log" Nov 03 18:02:24.367526 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:24.367429 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-njjrb_28008665-4e16-49dd-acc6-3e312861194d/main/0.log" Nov 03 18:02:24.754888 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:24.754860 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-m5d8l_a7aa56f7-e028-49dc-b1df-c6437a0096d6/main/0.log" Nov 03 18:02:28.032998 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:28.032956 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76"] Nov 03 18:02:28.045041 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:28.045004 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d5ae6a-bdn76"] Nov 03 18:02:28.788234 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:28.788197 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c93f4803-6fa1-4108-96a4-c2ea1faa7609" path="/var/lib/kubelet/pods/c93f4803-6fa1-4108-96a4-c2ea1faa7609/volumes" Nov 03 18:02:34.777869 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.777826 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mm7hs/must-gather-dpnwz"] Nov 03 18:02:34.778503 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.778480 6001 cpu_manager.go:414] "containerMap: removing container" podUID="fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" containerName="revoke-keys" Nov 03 18:02:34.778602 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.778504 6001 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" containerName="revoke-keys" Nov 03 18:02:34.778675 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.778646 6001 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd38b39-7f12-4eb0-87fc-be492ef9a0ee" containerName="revoke-keys" Nov 03 18:02:34.785139 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.785112 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:34.796617 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.796592 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/must-gather-dpnwz"] Nov 03 18:02:34.807108 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.807081 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"openshift-service-ca.crt\"" Nov 03 18:02:34.807108 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.807096 6001 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mm7hs\"/\"default-dockercfg-z6f5n\"" Nov 03 18:02:34.807302 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.807107 6001 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"kube-root-ca.crt\"" Nov 03 18:02:34.879728 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.879689 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92fff071-05e4-4382-9aab-755854192a50-must-gather-output\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:34.879728 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.879727 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdjbb\" (UniqueName: \"kubernetes.io/projected/92fff071-05e4-4382-9aab-755854192a50-kube-api-access-jdjbb\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:34.980966 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.980907 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92fff071-05e4-4382-9aab-755854192a50-must-gather-output\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:34.980966 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.980967 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdjbb\" (UniqueName: \"kubernetes.io/projected/92fff071-05e4-4382-9aab-755854192a50-kube-api-access-jdjbb\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:34.981288 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:34.981267 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/92fff071-05e4-4382-9aab-755854192a50-must-gather-output\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:35.005535 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:35.005504 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdjbb\" (UniqueName: \"kubernetes.io/projected/92fff071-05e4-4382-9aab-755854192a50-kube-api-access-jdjbb\") pod \"must-gather-dpnwz\" (UID: \"92fff071-05e4-4382-9aab-755854192a50\") " pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:35.101531 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:35.101410 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" Nov 03 18:02:35.413398 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:35.413365 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/must-gather-dpnwz"] Nov 03 18:02:35.413826 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:02:35.413801 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92fff071_05e4_4382_9aab_755854192a50.slice/crio-789615e498a158e24f1eccb414e5d3dd06477c053a343e00976c11a5a776b881 WatchSource:0}: Error finding container 789615e498a158e24f1eccb414e5d3dd06477c053a343e00976c11a5a776b881: Status 404 returned error can't find the container with id 789615e498a158e24f1eccb414e5d3dd06477c053a343e00976c11a5a776b881 Nov 03 18:02:35.416026 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:35.416004 6001 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:02:35.521406 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:35.521369 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" event={"ID":"92fff071-05e4-4382-9aab-755854192a50","Type":"ContainerStarted","Data":"789615e498a158e24f1eccb414e5d3dd06477c053a343e00976c11a5a776b881"} Nov 03 18:02:36.530160 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:36.530123 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" event={"ID":"92fff071-05e4-4382-9aab-755854192a50","Type":"ContainerStarted","Data":"16ea29f2a7ce804596c3cc0141a491b3ed7ec3c49f6ad011d5e28dddd79356d8"} Nov 03 18:02:36.530568 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:36.530173 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" event={"ID":"92fff071-05e4-4382-9aab-755854192a50","Type":"ContainerStarted","Data":"7eec52fda2c614041f1dfc4f3c8510019652b8e2cd4a3e59a11494ea1b245233"} Nov 03 18:02:36.575465 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:36.575011 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mm7hs/must-gather-dpnwz" podStartSLOduration=1.7726100900000001 podStartE2EDuration="2.574993301s" totalImagesPullingTime="802.383211ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:36.573733938 +0000 UTC m=+6451.906947538" watchObservedRunningTime="2026-11-03 18:02:36.574993301 +0000 UTC m=+6451.908206871" Nov 03 18:02:39.251250 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:39.251219 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4qqx2_a27eb936-8e3e-4bfb-8c6a-05c184ee89ea/global-pull-secret-syncer/0.log" Nov 03 18:02:40.273467 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:40.273191 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h2qrh_debced67-f8f8-480e-a1bc-4add3059fd0e/konnectivity-agent/0.log" Nov 03 18:02:40.502860 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:40.497270 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-130.ec2.internal_b90d8c1e0bb8232281e68f4371e9fed3/haproxy/0.log" Nov 03 18:02:47.119852 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:47.119786 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4c8d89d8-khtcc_7ac3d41f-22e1-4ef3-8389-4fc3e73d5196/authorino/0.log" Nov 03 18:02:48.271788 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:48.271755 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zd2c8_cb8bbc2c-4a30-45a4-a981-63996002b0c4/kuadrant-console-plugin/0.log" Nov 03 18:02:50.151484 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.151440 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/alertmanager/0.log" Nov 03 18:02:50.261066 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.261032 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/config-reloader/0.log" Nov 03 18:02:50.363991 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.363948 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/kube-rbac-proxy-web/0.log" Nov 03 18:02:50.432910 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.432822 6001 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h"] Nov 03 18:02:50.439212 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.439182 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.457716 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.457678 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h"] Nov 03 18:02:50.486466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.482177 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/kube-rbac-proxy/0.log" Nov 03 18:02:50.534391 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.534351 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-podres\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.534623 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.534454 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-sys\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.534623 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.534527 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-proc\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.534761 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.534627 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklxj\" (UniqueName: \"kubernetes.io/projected/718836a1-f09f-4d9a-8c87-62d8c70b10e2-kube-api-access-hklxj\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.534761 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.534655 6001 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-lib-modules\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.594275 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.594245 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/kube-rbac-proxy-metric/0.log" Nov 03 18:02:50.636013 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.635972 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-sys\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636023 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-proc\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636119 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-sys\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636133 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hklxj\" (UniqueName: \"kubernetes.io/projected/718836a1-f09f-4d9a-8c87-62d8c70b10e2-kube-api-access-hklxj\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636135 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-proc\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636232 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636201 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-lib-modules\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636535 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636277 6001 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-podres\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636535 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636312 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-lib-modules\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.636535 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.636429 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/718836a1-f09f-4d9a-8c87-62d8c70b10e2-podres\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.666120 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.666037 6001 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklxj\" (UniqueName: \"kubernetes.io/projected/718836a1-f09f-4d9a-8c87-62d8c70b10e2-kube-api-access-hklxj\") pod \"perf-node-gather-daemonset-vj99h\" (UID: \"718836a1-f09f-4d9a-8c87-62d8c70b10e2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.753899 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.753862 6001 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:50.762258 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.762234 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/prom-label-proxy/0.log" Nov 03 18:02:50.926621 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:50.926581 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7905790a-fc1a-4274-b8a5-cb1d685006e0/init-config-reloader/0.log" Nov 03 18:02:51.053421 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.053345 6001 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2"] Nov 03 18:02:51.065962 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.065928 6001 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01948d0b-464q2"] Nov 03 18:02:51.091538 ip-10-0-129-130 kubenswrapper[6001]: W1103 18:02:51.091494 6001 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod718836a1_f09f_4d9a_8c87_62d8c70b10e2.slice/crio-a146e51bde22a2ae65bdb33cece56e21a3acea108d7ef812291e4bb64fa87b67 WatchSource:0}: Error finding container a146e51bde22a2ae65bdb33cece56e21a3acea108d7ef812291e4bb64fa87b67: Status 404 returned error can't find the container with id a146e51bde22a2ae65bdb33cece56e21a3acea108d7ef812291e4bb64fa87b67 Nov 03 18:02:51.091900 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.091876 6001 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h"] Nov 03 18:02:51.227042 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.227009 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dg7h7_deeed625-5069-4da6-8482-2c8ca7c4ff88/kube-state-metrics/0.log" Nov 03 18:02:51.341004 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.340916 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dg7h7_deeed625-5069-4da6-8482-2c8ca7c4ff88/kube-rbac-proxy-main/0.log" Nov 03 18:02:51.456828 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.456798 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dg7h7_deeed625-5069-4da6-8482-2c8ca7c4ff88/kube-rbac-proxy-self/0.log" Nov 03 18:02:51.569437 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.569406 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68fbd467cc-f5zvj_efef1ee8-f44f-4b0f-9c5d-de4024cfb8da/metrics-server/0.log" Nov 03 18:02:51.675064 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.674928 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-n97fl_bc21144a-4ab2-4b38-b190-54a63a17c90c/monitoring-plugin/0.log" Nov 03 18:02:51.738236 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.738200 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" event={"ID":"718836a1-f09f-4d9a-8c87-62d8c70b10e2","Type":"ContainerStarted","Data":"8e8a5c5241231fc3585f5f1caa47667eb5c88cb3ff05e5230f6488a2d17d2ec9"} Nov 03 18:02:51.738466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.738262 6001 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" event={"ID":"718836a1-f09f-4d9a-8c87-62d8c70b10e2","Type":"ContainerStarted","Data":"a146e51bde22a2ae65bdb33cece56e21a3acea108d7ef812291e4bb64fa87b67"} Nov 03 18:02:51.738466 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.738402 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:51.788066 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:51.788011 6001 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" podStartSLOduration=1.7879959140000001 podStartE2EDuration="1.787995914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:51.785635257 +0000 UTC m=+6467.118848847" watchObservedRunningTime="2026-11-03 18:02:51.787995914 +0000 UTC m=+6467.121209482" Nov 03 18:02:52.138806 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.138766 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9q8w_caa2b183-e4e8-41fc-85e4-52293e0add64/node-exporter/0.log" Nov 03 18:02:52.250626 ip-10-0-129-130 kubenswrapper[6001]: E1103 18:02:52.250581 6001 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-phsml" containerName="extractor" Nov 03 18:02:52.258476 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.258433 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9q8w_caa2b183-e4e8-41fc-85e4-52293e0add64/kube-rbac-proxy/0.log" Nov 03 18:02:52.436967 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.436891 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n9q8w_caa2b183-e4e8-41fc-85e4-52293e0add64/init-textfile/0.log" Nov 03 18:02:52.542394 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.542365 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-kgglr_601ebbab-626e-4679-a41e-437cdf951a1c/kube-auth-proxy/0.log" Nov 03 18:02:52.791820 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.791772 6001 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6566161b-4221-4cb2-b878-10ba73c722f8" path="/var/lib/kubelet/pods/6566161b-4221-4cb2-b878-10ba73c722f8/volumes" Nov 03 18:02:52.801229 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.801195 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-bgct8_90e58ef7-2fe2-4060-af83-3d06ba7cbea9/istio-proxy/0.log" Nov 03 18:02:52.898851 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.898822 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj6xz_a13a995e-bff6-4420-b8d3-4d2f107c2fa4/kube-rbac-proxy-main/0.log" Nov 03 18:02:52.913012 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:52.912977 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fmxr5_c9732c01-e962-45c8-a46c-c83c88a65a4e/istio-proxy/0.log" Nov 03 18:02:53.002725 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.002666 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj6xz_a13a995e-bff6-4420-b8d3-4d2f107c2fa4/kube-rbac-proxy-self/0.log" Nov 03 18:02:53.084008 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.083922 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xsc9j_7b971155-38f4-436e-b6a8-b7224b30286e/istio-proxy/0.log" Nov 03 18:02:53.104185 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.104155 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qj6xz_a13a995e-bff6-4420-b8d3-4d2f107c2fa4/openshift-state-metrics/0.log" Nov 03 18:02:53.216408 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.216382 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/prometheus/0.log" Nov 03 18:02:53.316631 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.316584 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/config-reloader/0.log" Nov 03 18:02:53.318994 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.318957 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d85d667ff-kc82s_9139f8c8-371a-44d5-8d0e-636f0eb11e48/payload-processing/0.log" Nov 03 18:02:53.427121 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.427013 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/thanos-sidecar/0.log" Nov 03 18:02:53.529490 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.529414 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/kube-rbac-proxy-web/0.log" Nov 03 18:02:53.634097 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.634070 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/kube-rbac-proxy/0.log" Nov 03 18:02:53.735502 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.735470 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/kube-rbac-proxy-thanos/0.log" Nov 03 18:02:53.835793 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:53.835761 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d169bc56-6a2e-4d13-a438-32dbfd59d915/init-config-reloader/0.log" Nov 03 18:02:54.663681 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:54.663648 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/thanos-query/0.log" Nov 03 18:02:54.806401 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:54.806369 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/kube-rbac-proxy-web/0.log" Nov 03 18:02:54.913944 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:54.913833 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/kube-rbac-proxy/0.log" Nov 03 18:02:55.014189 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:55.014162 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/prom-label-proxy/0.log" Nov 03 18:02:55.116654 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:55.116627 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/kube-rbac-proxy-rules/0.log" Nov 03 18:02:55.217154 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:55.217075 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5765448457-qf2dp_4bdfec9a-b17a-4fc9-bcb1-2b0be3a97c54/kube-rbac-proxy-metrics/0.log" Nov 03 18:02:57.380201 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:57.380166 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6d4c8d89d8-khtcc_7ac3d41f-22e1-4ef3-8389-4fc3e73d5196/authorino/0.log" Nov 03 18:02:57.534674 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:57.534644 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-zd2c8_cb8bbc2c-4a30-45a4-a981-63996002b0c4/kuadrant-console-plugin/0.log" Nov 03 18:02:57.762043 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:57.761099 6001 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vj99h" Nov 03 18:02:58.527129 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:02:58.527093 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-559f9fb54-tnqtx_74a03a30-b6cf-4f1f-8560-1ab5ca56647e/console/0.log" Nov 03 18:03:01.935755 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:01.935726 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2dsmp_26796971-9faf-4381-8f8b-23db59a37fcc/dns/0.log" Nov 03 18:03:02.057576 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:02.057547 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2dsmp_26796971-9faf-4381-8f8b-23db59a37fcc/kube-rbac-proxy/0.log" Nov 03 18:03:02.730429 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:02.730394 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t6s9c_f6cd40ea-b486-4d7d-a8ec-49d153976a09/dns-node-resolver/0.log" Nov 03 18:03:04.159584 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:04.159552 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4tjbw_a163d0db-7c08-45c9-bf83-6132d280a72a/node-ca/0.log" Nov 03 18:03:05.670536 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:05.668793 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-kgglr_601ebbab-626e-4679-a41e-437cdf951a1c/kube-auth-proxy/0.log" Nov 03 18:03:05.957331 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:05.955708 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-bgct8_90e58ef7-2fe2-4060-af83-3d06ba7cbea9/istio-proxy/0.log" Nov 03 18:03:06.118548 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:06.118518 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fmxr5_c9732c01-e962-45c8-a46c-c83c88a65a4e/istio-proxy/0.log" Nov 03 18:03:06.363778 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:06.363746 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xsc9j_7b971155-38f4-436e-b6a8-b7224b30286e/istio-proxy/0.log" Nov 03 18:03:06.629352 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:06.629278 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6d85d667ff-kc82s_9139f8c8-371a-44d5-8d0e-636f0eb11e48/payload-processing/0.log" Nov 03 18:03:08.530457 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:08.530421 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-kgglr_601ebbab-626e-4679-a41e-437cdf951a1c/kube-auth-proxy/0.log" Nov 03 18:03:08.967021 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:08.966906 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-bgct8_90e58ef7-2fe2-4060-af83-3d06ba7cbea9/istio-proxy/0.log" Nov 03 18:03:09.208738 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:09.208698 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-fmxr5_c9732c01-e962-45c8-a46c-c83c88a65a4e/istio-proxy/0.log" Nov 03 18:03:09.544642 ip-10-0-129-130 kubenswrapper[6001]: I1103 18:03:09.544577 6001 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-xsc9j_7b971155-38f4-436e-b6a8-b7224b30286e/istio-proxy/0.log"